HSS/HSF/DP.07.5
Assessing the Reliability of Household Expenditure Data: Results of the World Health Survey
DISCUSSION PAPER NUMBER 5 - 2007
Department "Health System Financing" (HSF) Cluster "Health Systems and Services" (HSS)
World Health Organization 2007 © We thank Somnath Chatterji for his valuable comments in the early draft. The authors are also grateful to Nirmala Naidoo for her assistance on various questions related to the procedure in conducting the World Health Survey. The authors also benefited from the discussion in the Health Systems Financing Department seminar held in WHO and the iHEA 2007 Congress. Last but not the least we would like to acknowledge Chris Murray for his inspiration to us in exploring this topic. The views expressed in documents by named authors are solely the responsibility of those authors.
Assessing the Reliability of Household Expenditure Data: Results of the World Health Survey
by Ke Xu, Frode Ravndal, David Evans & Guy Carrin
GENEVA 2007
3
Abstract The World Health Survey (WHS) which has been implemented in more than 70 countries with standardized questionnaires opens a great opportunity for research on health care financing issues. This study examines the household expenditures and health expenditure collected in the WHS in terms of reliability, consistency between different ways of data collection within the survey and with other types of household surveys. Data used in this study include 50 WHS and 37 other type of surveys, namely the Living Standard Measurement Survey, Household budget Survey and Income and Expenditure Survey. The analysis consists of comparison of test-retest results; the aggregated and reported total household expenditure and health expenditure; the expenditures from the WHS and other type of surveys. The results from test-retest are fairly similar in the WHS. For health expenditure the average of reported total is lower than the aggregated total while for household total expenditure the estimate is fairly similar from the two measures. Finally the WHS was found to report lower total household expenditure but higher out-of-pocket expenditure comparing with other types of surveys. The study suggests further efforts to standardize the questions in collecting expenditure data in household surveys for the purpose of cross country and over time comparison.
4
Introduction Household expenditure data has been used extensively for monitoring general household living standards, wellbeing and consumption patterns.(1) More recently, considerable attention has been paid to monitoring household expenditures on health with a view to determining if the need to pay for services prevents some people from seeking or continuing care, and results in severe financial hardship or impoverishment for others (2-4). This literature has grown considerably over the last five years, with analysts using expenditure data from whatever source they can find, including the Living Standard Measurement Survey (LSMS) supported by the World Bank, Household Budget Surveys (HBS), Income and Expenditure Surveys (IES) and Socio-economic Surveys (SES)(5-8). There has long been concern with the accuracy of expenditure data reported in household surveys, often linked to concerns about the abilities of households to remember a multitude of different types of expenditures accurately(9-11). Measurement error can be introduced at any stage of a survey: design of the survey instrument, data collection, or data entry(12). This is partly because household expenditure surveys are among the most difficult and expensive surveys to field and are sometimes undertaken with less than sufficient funding (13). While these concerns are well established, there has been little attempt to understand the extent to which phrasing questions in different ways can influence the response to health expenditure questions, and whether different types of surveys produce consistent results. We contribute to this literature by comparing two ways of seeking information on health expenditure developed in the World Health Survey (WHS), and then also consider the extent to which the estimated expenditures are consistent with expenditure derived from other surveys undertaken in the same countries at approximately the same time. The WHS were launched by the World Health Organization to strengthen national capacity to monitor critical health inputs, outputs and outcomes (14). They collected information on total household expenditure with a breakdown that included health expenditures, together with a wide range of indicators on health status, health service utilization, risk factors, and the perceived responsiveness of the health system. This makes the WHS appealing to policy makers and researchers seeking information on diverse topics including the assessment of inequality of health and in intervention coverage across different socio-economic groups. World Health Surveys have been implemented in 72 countries using standard questionnaires and many of the country data sets have recently been put into the public domain (http://www.who.int/healthinfo/survey/en/index.html ).
Methodology Instrument used in the WHS The World Health Surveys currently available for analysis were conducted in 72 countries during 2002 and 2003. All are nationally representative using a multistage stratified random cluster sampling strategy. Data were collected at both the household and individual level. Among the 72 countries, 50 used the so-called long version household questionnaire (applied only in low and middle income countries) which gives details of the breakdown of total household expenditure and out-of-pocket health expenditure into their different categories.
5
The expenditure data were collected at the household level from the selected household informant. The questionnaire first seeks information on total household expenditure over the last month, and then asks details of item-by-item expenditure over the same period. The specified items are food, housing, education, health care, voluntary health insurance premiums, and all other goods and services. Respondents are asked to report on both cash and in kind payments. Health expenditure excludes transportation cost to obtain care and is net of insurance reimbursement. At another point in the survey, to check consistency, respondents are asked to provide item-by-item details of their health expenditures. In this case, the listed items are inpatient care, outpatient care, traditional medicine, dentists, medication or drugs, health care products, laboratory tests, and all other health care products or services. The initial plan was that test-retests would be undertaken for a minimum of 10% of the sample in all countries conducting the World Health Survey. However, not may countries met the request. We therefore examine test-retest reliability for all surveys that reached the 10% sample target, and who retested more than 100 households. Twenty-four out of the fifty countries met these criteria. Retests were conducted within a week of the initial interview. Other data sources used in the analysis Thirty seven of the countries that have implemented the WHS had also conducted other types of household surveys with questions on total and health expenditure sometime during the period after 1990. The survey instruments differed and details are found in appendix 1, but they included Living Standards Measurement Surveys (LSMS), Household Income and Expenditure Surveys (IES), Household Budget Surveys (HBS) and Socio-economic Surveys (SES). The LSMS and the SES are multi-purpose surveys where the expenditure module is an important component. The detail sought in the expenditure breakdowns and the recall periods varied by country, but in most cases, more breakdown items on household general expenditure were employed than in the WHS. For health expenditure, the number of questions in the comparator surveys ranged from one to as many as those in the WHS. Recall period also varied in these surveys. Typically a one-month recall period was used for frequent spending and a one-year recall period for durables, sometimes including hospitalization. The IES and HBS asked for a more detailed breakdown of health expenditures than the LSMS and SES. Analysis framework Reliability refers to the repeatability or consistency of a set of measurements or measuring instrument (15). A measure is considered reliable if it would give us the same result over and over assuming that what we are measuring isn't changing. Reliability could be characterized as either internal or external. Internal reliability is a measure of internal consistency. It compares two sets of data on the same subject using differnt meansures. External reliability means the extent to which data measured at one time is consistent with data from the same variable measured at another time. The test-retest technique is commonly used to examine external reliability(16;17). For internal reliability we compared the difference between the total reported in response to the single question and the total derived by aggregating responses to the questions asking for components of expenditure - called the "reported" and "aggregated" totals respectively. The test-retest information is used to examine external reliability The intra-class coefficient index
6
(ICC) was used to explore both types of reliability and it was applied to the responses to total household expenditure and household health expenditures (18). The ICC is calculated as
σ 2 (b) ICC = 2 σ (b) + σ 2 ( w) where σ2(w) is the pooled variance of a variable between survey administrations, and σ2(b) is the variance of the same variable between subjects (respondents). The ICC is interpreted as the proportion of total variance accounted for by between-subject or between-question variation. When there is no variance between the two administrations the value is 1. Furthermore, the study compared the expenditure estimates produced by the WHS and the other types of household surveys undertaken in the same countries. The comparisons include food expenditure, total household expenditure and health expenditure, as well as the shares of food and health expenditure in total household expenditure. GDP deflators are used to convert the value from the survey years to the year 2000. Household sampling weights, where available, are used to account for differential probabilities of selection, and to ensure comparability across surveys. Results Results from test-retest in the WHS Figure 1 reports the ICCs for the test-retest responses for total household expenditure and expenditures on education, food and health. Each vertical bar depicts a country, and the range shows the 95% confidence intervals around the mean estimate of the ICC. For most countries, the average value of the ICC is above 0.6 for all items, which is generally considered to imply good external reliability (19;20). The lowest for household expenditure is 0.28, for food 0.19, for education 0.39 and for total out-of-pocket health expenditure 0.22. Some countries have very high test-retest ICCs for all items, suggesting high consistency, examples are Sri Lanka, Myanmar, China, Uruguay, Malaysia and Pakistan. On the other hand, the average ICCs were consistently lower than 0.5 in Nepal and the Dominican Republic. Insert figure 1
2. Comparison of the reported total and aggregated total expenditure in the WHS Details of the ICC index in reported and aggregated total are found in Figure 2 where, again, each vertical bar represents a different country and ranges depict the 95% confidence interval around the mean estimate. For total household expenditure, the ICC is above 0.5 for all 50 countries with four exceptions - Mauritania, Zimbabwe, Ghana and Ecuador. For health expenditure the ICC index is lower than 0.5 only in 6 countries: Mauritania, Zambia, Uruguay, Swaziland, Kenya and Czech Republic. The band for total expenditure is much narrower than for health expenditures. This is mainly explained by the fact that there were less zero values or non-reports to the questions on total expenditures than on health expenditures.
7
Inset figure 2 The lack of consistency between the reported and the aggregated total is partly the result of some people reporting zero values to the question on the reported total yet then reporting positive expenditures to some of the components at a later point, or the other way round. Figure 3 showed in nearly all countries that more households responded to questions on breakdown items than to the reported total question on household expenditure. However, for health expenditure the results are not consistent across countries. Inset figure 3 While this is important in itself, we also considered what happened in the non-zero cases by comparing the ratio of the reported total to the aggregated total. The average household total expenditure is similar between the two measures, with the difference never exceeding 20% except in the case of Ecuador (figure 4-a). For health expenditure, the average reported total across all respondents is smaller than the average aggregated total in all countries except Ecuador and Uruguay, in most cases by a substantial margin (figure 4-b). However, because expenditure data rarely conform to a normal distribution, averages are sensitive to extreme values. To check if extreme values are driving these results, we also compared the ratios at different percentiles of expenditure: 5th, 25th, 50th, 75th and 95th. Figures 4-c and d show the results for both total health expenditure and total household expenditures in the two panels. Each box presents 50% of the observations with the upper hinge the 75th percentile, the lower hinge set at the 25th percentile and the bar showing the median. Even though there is some variation in the ranges across the different deciles, there is no clear evidence that outliers are driving the results. However, the reported total gave a higher estimate than aggregated value in the lowest 5th percentile in heath expenditure and slightly higher estimate in household total expenditure. This can be explained by the fact that a small number in the reported total reflects the sum of total spending while a small number in the aggregated total may only come from one item. There is no way to know whether other items are missing or zero. Insert figure 4 3. Differences between the WHS and other surveys Health expenditure and food expenditures, in absolute terms and as a share of household total expenditure, derived from the WHS were compared with the same variables derived from other types of surveys where this was possible - i.e. in 37 countries. Figure 5 presents the results for the shares of food and health in total household expenditure. In Figures 5 a and b, the horizontal axis represents the WHS estimate, and the vertical axis represents the estimate from the comparator survey. The diagonal line shows that the points at which the estimates would be identical. The estimated share of health in total expenditure is consistently higher in the WHS (figure 4a), with the exception of three countries where they give similar results. A similar pattern is observed for the share of food in total expenditures, with the exceptions of Kazakhstan, Laos and Comoros where the WHS suggests slightly higher shares (figure 5-b). The average share
8
of health expenditure in the 37 countries is 6.9% (ranging from 1.5-12.6%) in the WHS compared to 3.4% (ranging form 0.4-9.8%) in the other surveys, while the average food share is 58% (ranging from 42-74%) in the WHS and 51% (ranging from 25-78%) in the other surveys. Insert figure 5 A higher proportion of health spending in total expenditure could be due to two things - health spending could be higher, and/or other spending could be lower. We explore this in Figure 6. In absolute terms, the average total household expenditure and average food expenditure derived from the WHS are both smaller than those derived from the other surveys in most countries (Figure 6-a). In addition, average health spending is higher in the WHS than in the other surveys in most countries, with 11 exceptions. Accordingly, in general, though not always, estimates of health spending from the WHS are higher than those derived from other surveys and that estimates of non-health spending are lower. Further comparison by percentiles finds that in all selected 6 percentiles the household expenditure derived from the WHS is lower than those derived from the other surveys in most countries (Figure 6-b). The comparison on the food expenditure shows similar pattern across all percentiles, except the 5th percentiles which shows more variability than for people who spend more (Figure 6-c). For health, over 20% of households typically reported zero expenditures, which accounts for the inability to compare the responses for the 5th and 25th percentiles (Figure 6-d). In the other cases, there is considerable variation in the ratio with some evidence that outliers might be important for the 75% percentile. Insert figure 6 Discussion The WHS will be a major source for health and health system related studies. In the area of health financing studies, including out-of-pocket health expenditure, financial catastrophe and impoverishment by health payment, the WHS has great potential to fill in the gaps where no appropriate household surveys exist or where the existing surveys are not up to date. Information on quality of the data is crucial for researchers in analysing the data and interpreting the results. Are expenditure data in the WHS reliable? The results from test-retest reliability shows that the ICC is high in most countries. Apart from the reliability of the data the ICC is also influenced by the length of the interval between the two administrations. A short interval between administrations of the instrument will tend to yield too high reliability due to learning. Obviously a too long interval will lead to a low ICC as the spending has changed. In the WHS the interval between the two administrations is one week, ensuring that the two administrations are approximately comparable. Unlike constant variables such as sex, we do not expect the ICC in expenditure to reach 1. The interval between the two administrations is one week. So it is possible that the numbers in the test and retest data are different. Do long expenditure questionnaires give a higher estimate? Shorter questionnaires has lower survey costs compared to longer ones, while the longer ones seem to give more accurate estimates. However, it is not always true that the longer the questionnaires the more accurate the numbers obtained. In general the longer the
9
questionnaires the higher the estimates (21-24). In the WHS, no significant differences are observed in household total expenditure between reported and aggregated total from the 6 breakdown items. For health expenditure the aggregated total is greater that the reported total which is coherent with the literature. One important variable in health financing research is health expenditure as a share of total household consumption. The study suggests that when using the WHS it is more appropriate to use the aggregated household expenditure (6 items) and the reported total health expenditure in order to estimate the share of health in total household expenditure. Still, it is obvious that the breakdown items on health give more information when studying the components of health spending. Does the WHS overestimate health expenditure and underestimate other household expenditures? The differences between any two surveys are expected because of the different survey years, survey designs and the different recall period(25). However the WHS does give higher estimates for health expenditure even compared with the reported total health spending, and yields lower estimates on food expenditure and other expenditures. There could be several reasons. The most important factor is the survey design. The WHS is an intensive health focused survey. In such a situation the respondent may include spending on health that took place earlier than the past month, which would cause an upward bias. By the same token other expenditures in such health focused surveys may be subject to a downward bias. The recall period could also contribute to the difference between the WHS and other types of surveys in health expenditure and other expenditure. In the WHS the recall period is one month for all expenditure items, while in other surveys various recall periods were used Longer recall period may increase recall bias, but meanwhile it can capture more infrequent spending. The overall effect is not clear. The length of the expenditure section questionnaires is another factor which can contribute to the difference. The WHS has much shorter questionnaires for household expenditure items than other types of surveys in. This may also account in part for the fact that food and other expenditures are lower in the WHS. This however, does not apply to health expenditure. Finally, the WHS was conducted during 2002 to 2003 while the other surveys were conducted in earlier years. It could be that in some countries household total spending was reduced while health spending increased. However, this did not happen in all countries and comparing the differences between the WHS and other types of survey, the real changes in the expenditure pattern is trivial. Conclusion The WHS has great applicability to a range of health care financing studies. Countries need timely information to evaluate their health policies, manage their health systems and monitor progress. The WHS may be best viewed as another source of survey data to supplement the information provided by routine national information systems.
10
In this study we found that the expenditure data in the WHS are reliable based on the testretest estimates. The aggregated total gives higher non-zero response rate than reported total in household total expenditure, but this can not be generalized to health expenditure. Furthermore, the average estimates from the two ways of asking questions yield similar results in household total expenditure. However, for health expenditure the aggregated total exceeds the reported total. The results suggest that the intensive health focused WHS tends to give a higher estimate in health expenditure but a lower estimate in other expenditures. While the WHS is a good source for cross-country comparison studies, we need to be cautious with comparative studies using other types of surveys on household total expenditure and health expenditure. Finally, the study also proposes that standardizing the questionnaires in collecting household expenditure data would be beneficial in order to better conduct comparative studies across countries and over time.
11
APPENDICES APPENDIX 1 country United Arab Emirates Burkina Faso Bangladesh Bosnia and Herzegovina Brazil China Côte d'Ivoire Congo Comoros Czech Republic Dominican Republic Equador Spain Estonia Ethiopia Georgia Ghana Croatia India Kazakhstan Kenya Lao People's Dem. Rep. Sri Lanka Latvia Morocco Mexico Mali Myanmar Mauritania Mauritius Malawi Malaysia Namibia Nepal Pakistan Philippines Paraguay Russian Federation Senegal Slovakia Slovenia Swaziland Chad Tunisia Ukraine
Data used in the analysis (50 countries) code ARE BFA BGD BIH BRA CHN CIV COG COM CZE DOM ECU ESP EST ETH GEO GHA HRV IND KAZ KEN LAO LKA LVA MAR MEX MLI MMR MRT MUS MWI MYS NAM NPL PAK PHL PRY RUS SEN SVK SVN SWZ TCD TUN UKR survey name World Health Survey World Health Survey Enquête Prioritaire sur les Conditions de Vie des Ménages World Health Survey Household Expenditure Survey World Health Survey World Health Survey LSMS World Health Survey World Health Survey World Health Survey World Health Survey World Health Survey Household Budget Survey World Health Survey World Health Survey World Health Survey Encuesta Continua de Presupuestos Familiares World Health Survey Household Budget Survey World Health Survey World Health Survey National Household Revenue and Expenditure Survey World Health Survey Ghana Living Standards Survey World Health Survey World Health Survey World Health Survey LSMS World Health Survey World Health Survey Lao Expenditure and Consumption Survey II (LECS II) World Health Survey Household Income and Expenditure Survey World Health Survey Household Expenditure Survey World Health Survey Enquêtes sur les conditions de vie des ménages World Health Survey Encuesta Nacional de Ingresos y Gastos World Health Survey World Health Survey World Health Survey World Health Survey Household Expenditure Survey World Health Survey Integrated Household Survey World Health Survey Household Expenditure Survey World Health Survey Household Income and Expenditure Survey World Health Survey LSMS World Health Survey Pakistan Integrated Household Survey World Health Survey Family Income and Expenditures Survey World Health Survey Encuestas de Hogares World Health Survey World Health Survey Enquête Sénégalaise auprès des ménages (ESAM) World Health Survey Family Expenditure Survey World Health Survey World Health Survey World Health Survey World Health Survey L’enquête Nationale sur le Budget et la Consommation des Ménages World Health Survey Income Expenditure Survey type WHS WHS LSMS WHS HES WHS WHS LSMS WHS WHS WHS WHS WHS HBS WHS WHS WHS Other WHS HBS WHS WHS IES WHS LSMS WHS WHS WHS LSMS WHS WHS HES WHS IES WHS HES WHS LSMS WHS IES WHS WHS WHS WHS HES WHS LSMS WHS HES WHS IES WHS LSMS WHS LSMS WHS IES WHS LSMS WHS WHS Other WHS HES WHS WHS WHS WHS HBS WHS IES year 2003 2003 1998 2003 1996 2003 2003 1996 2003 2003 2003 2003 2003 1999 2003 2003 2003 1996 2003 1995 2003 2003 1999 2003 1999 2003 2003 2003 1996 2003 2003 1997/98 2003 1995/96 2003 1997/98 2003 1991 2003 1996 2003 2003 2003 2003 1996 2003 1997/8 2003 1993/94 2003 1994 2003 1995/96 2003 1991 2003 1997 2003 1996 2003 2003 1994/95 2003 1993 2003 2003 2003 2003 1995 2003 1996 sample size 1169 4930 8476 5932 7420 841 4961 4850 3991 3160 2889 1831 807 2675 4950 4521 5685 3104 994 2818 4274 2754 2846 4139 5998 988 10548 4497 1994 4594 4971 8881 6777 19631 881 7684 4996 2574 38483 13661 4242 6045 3749 3962 6233 5488 9118 6083 14628 4249 4384 8790 3373 6440 4771 10072 39520 5268 2588 3631 3349 3274 1756 2129 660 2801 4785 5118 5140 2613 2272
12
Uruguay Viet Nam South Africa Zambia Zimbabwe
URY VNM ZAF ZMB ZWE
World Health Survey Encuesta de Gastos e Ingresos de los Hogares World Health Survey Vietnam Living Standard Survey World Health Survey South Africa Income Expenditure Survey World Health Survey Living Conditions Monitoring Survey World Health Survey
WHS IES WHS LSMS WHS IES WHS LSMS WHS
2003 1994/5 2003 1992/93 2003 1995 2003 1996 2003
2971 3748 4171 4799 2378 29594 4157 11073 4144
13
Figure 1. The intra-class coefficient (ICC) for test-retest Education 1
Total household expenditure
ICC
0
.2
.4
.6
.8
Food 1
Health
0 0
.2
.4
.6
.8
.2
.4
.6
.8
1
0
.2
.4
.6
.8
1
ICC
Figure 2. The intra-class coefficient (ICC) for reported total and breakdown total Household total expenditure 1
Health expenditure
ICC .2 .4
.6
.8
ECU MRT
0 0
.2
.4
.6
.8
1
0
.2
.4
.6
.8
1
ICC
14
Figure 3. Fraction of reported zero values in household total expenditure and health expenditure Household total expenditure .8
Health expenditure
.2
fraction
.15
fraction 0 10 20 30 40 50
.1
.05
0
0 0
.2
.4
.6
10
20
30
40
50
countries reported aggregated
countries reported aggregated
Figure 4. Comparison of reported total and breakdown total Household total expenditure, mean-(a) 0 .2 .4 .6 .8 1 1.21.41.6 0 .2 .4 .6 .8 1 1.21.41.6
Health expenditure, mean-b) reported/aggregated
reported/aggregated
0
10
20
30
40
50
0
10
20
30
40
50
countries Household total expenditure, percentile-(c) .4 .6 .8 1 1.2 1.4 1.6 .4 .6 .8 1 1.2 1.4 1.6
countries Health expenditure, percentile-(d) reported/aggregated
reported/aggregated
5th
25th
50th
75th
95th
5th
25th
50th
75th
95th
15
Figure 5. Food and health expenditure as a share of total household expenditure (WHS vs. other surveys) Health expenditure share-(a) .14 .8
food share-(b)
.12
LAO GEO BGD
MWI
.1
KAZ
PAK
.6
NPL BFA ZMB IND GHA LKA VNM UKR
other surveys .06 .08
other surveys
BRA CIV
CHN
RUS CIV PRY KEN MAR
PHL SEN PAK
VNM
BRA MUS LVA
NAM TUN
PRY
.4
CHN IND
EST MYS SVK SVNHRV
.04
URY LVA KEN MEX
RUS GHA EST SEN MUS LKA NPL GEO BFA
MEX
TUN UKR MAR LAO KAZ BGD
ZAF
.02
ZMB SVN
ESP
CZE HRV MYS NAMMWI ZAF SVK
PHL
CZE URY
ESP
0
.02
.04
.06 WHS
.08
.1
.12
.14
.2 .2
0
.4 WHS
.6
.8
Figure 6. Total household expenditure, food and health expenditure in absolute terms (WHS over other surveys) Mean-(a) 0 .5 1 1.5 2 2.5 3 WHS/other survey 1 1.5 .5 household total expenditure-(b)
WHS/other survey
Total exp
food
oop
0 5th
25th
50th
75th
95th
food expenditure-(c) 1.5 WHS/other survey 1 2 3 4 5 6
health expenditure-(d)
WHS/other survey
0
.5
1
0 5th
5th
25th
50th
75th
95th
25th
50th
75th
95th
16
Reference List
1. Deaton A, Case A. Analysis of Household Expenditures. World Bank; 1988. 2. Su T, Kouyaté B, Flessa S. Catastrophic household expenditure for heath care in a low income Society: a study from Nouna District, Burkina Faso. Bulletin of the World Health Organization 2006;84(1):21-7. 3. Xu K, Evans DB, Kadama P, Nabyonga J, Ogwal PO, Nabukhonzo P et al. Understanding the impact of eliminating user fees: Utilization and catastrophic health expenditures in Uganda. Social Science and Medicine 2006;62(4):866-76. 4. Devadasan N, Van Damme W, Criel B, Ranson K, Van der Stuyft P. Indian community health insurance schemes provide partial protection against catastrophic health expenditure. BMC Health Services Research 2007;7(43). 5. Habicht J, Xu K, Couffinhal A, Kutzin J. Detecting changes in financial protection: creating evidence for policy in Estonia. Health Policy Plan. 2006;21(6):421-31. 6. Van Doorslaer E, O'Donnell O, Rannan-Eliya RP, Somanathan A, Adhikari SR, Garg CC et al. Effect of payments for health care on poverty estimates in 11 countries in Asia: an analysis of household survey data. Lancet 2006;368(9544):1357-64. 7. Wagstaff A, Van Doorslaer E. Catastrophe and impoverishment in paying for health care: With applications to Vietnam 1993-1998. Health Economics 2003;12(11):921-34. 8. Xu K, Evans DB, Kawabata K, Zeramdini R, Klavus J, Murray CJL. Household catastrophic health expenditure: A multicountry analysis. Lancet 2003;362(9378):111-7. 9. Raphael Branch E. The Consumer Expenditure Survey: a comparative analysis. Monthly Labor Review 1994;117(12):47-55. 10. Visaria P. Poverty and Living Standards in Asia. Population and Development Review 1980;6(2):189-223. 11. Anand S, Harris CJ. Choosing A Welfare Indicator. American Economic Review 1994;84(2):226-31. 12. Neter J. Measurement Errors in Reports of Consumer Expenditures. Journal of Marketing Research 1970;7(1):11-25. 13. Mcwhinne, I., Champion HE. Canadian Experience with Recall and Diary Methods in Consumer Expenditure Surveys. Annals of Economic and Social Measurement 1974;3(2):411-&.
17
14. Üstün B, Chatterji S, Villanueva M, Bendib L, Çelik C, Sadana R. WHO Muliti-country Survey Study on Health and Responsiveness 2000-2001. In: Murray C, Evans D, editors. Health Systems Performance Assessment: Debates, Methods and Empiricism. Geneva: World Health Organization; 2003. 15. Murray C. Towards good practice for health statistics: lessons from the Millennium Development Goal health indicators. The Lancet 2007;369(9564):862-73. 16. Bland JM, Altman DG. Statistics notes: Measurement error proportional to the mean (vol 313, pg 106, 1996). British Medical Journal 1996;313(7059):744. 17. Bland JM, Altman DG. Measurement error and correlation coefficients. British Medical Journal 1996;313(7048):41-2. 18. Muller R, Buttner P. A Critical Discussion of Intraclass Correlation-Coefficients. Statistics in Medicine 1994;13(23-24):2465-76. 19. Hume C, Ball K, Salmon J. Development and reliability of a self-report questionnaire to examine children's perceptions of the physical activity environment at home and in the neighbourhood. International Journal of Behavioral Nutrition and Physical Activity 2006;3(1):16. 20. Sim J, Wright C. Research in Health Care: Concepts, Designs and Methods. Cheltenham: Stanley Thornes Ltd; 2000. 21. Jolliffe, D. and Scott, K. The sensitivity of measures of household consumption to survey design: results from an experiment in El Salvador. 1995. Washington DC., World Bank. 22. Steele, D. Equador consumption items. Internal memorandum. 1998. Washington DC, Development Research Group. World Bank. 23. Reagan B. Condensed versus detailed schedule for collecting of family expenditure data. Agricultural Research Service. US Department of Agriculture.; 1954. 24. Browning N, Crossley TF, Weber G. Asking consumption questions in general purpose surveys. Economic Journal 2003;113(491):F540-F567. 25. Grosh, M. and Glewwe, P. Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study. 1. 2000. Washington DC, World Bank.
18