Meta-analysis and selection of preferred indicators

4. Meta-analysis and selection of preferred indicators Introduction Results are reported in Chapters 5 and 6, by out- come, in the following order: fetal outcomes-low birth weight, intrauterine growth retardation, and preterm birth; maternal outcomes-complications of delivery, pre-eclampsia, and postpartum haemorrhage. For each outcome, a summary table gives the combined odds ratio (OR) for each indicator in the order (and abbreviation) as given in Chapter 1, Table 3. This is followed by a more detailed review of selected individual indicators with graphs of indica- tor profiles (Fig. 4 is an example). Interpretation of the profile plot The centre of each indicator profile (Fig. 4) identifies the estimated mean OR for that indicator. As the curve is a probability plot, the width of each profile indicates the spread of values about the mean. This provides a visual check on the degree of uncertainty associated with the estimate. The profile plot also clearly identifies the similarities or differences be- tween a set of related indicators-whether there is a strong overlap between these or whether one or more indicators stand apart. The ORs are calculated for weight below the lowest quartile versus weight Fig. 4. Odds ratios for combined profiles following meta-analysis for IUGR by pre-pregnancy weight and weight attained by months 5, 7 and 9. Combined profile, pre-pregnancy Combined profile, month 5 (lunar) mont Combined profile, month 7 (lunar) Combined profile, month 9 (lunar) 1.0 1.75 2.5 3.25 4.0 Odds ratio above the highest quartile.a In this plot the profile for pre-pregnancy weight is clearly separated from the pair of profiles for attained weight at months 7 and 9. Attained weight at months 7 and 9 overlap consi- derably and a formal test would confirm no signifi- cant difference between the means (ORs: 3.03 and 3.09, respectively) for these two distributions. The profile for month 5 falls midway between both sets (OR: 2.77). The profiles for months 5, 7 and 9 are relatively broad, indicating that the mean OR is less precisely estimated in these, compared with the higher degree of precision (i.e., narrower profile) for estimates of the mean OR for pre-pregnancy weight. Test for homogeneity In examining the summary tables (see Chapter 3, Tables 7-9, pp. 15-17), it is important to recall the caveat noted in Chapter 2 in relation to the estimate of the odds ratio for the combined profiles. This is based on the 'face value' analysis alone, the assump- tion being that there is a single true value for the parameter and that the observed differences in the estimates are due simply to random error. A test for homogeneity of results across studies will reveal whether this is the case. If we are prepared to accept the evidence at face value, then this summary table provides the simplest indication of the expected magnitude of the odds ratios for each indicator for the given outcome. If there is evidence of lack of homogeneity in the results for a given indicator, a weighted regres- sion analysis is normally conducted in which an a This group has been selected for several practical reasons. First, there is reasonable stability in the parameter estimates. For the potentially higher risk group-those below the 10th per- centile of the relevant anthropometric distribution-it was found that the stability of the parameter estimates was affected by the (often) very small numbers available for modelling in this catego- ry. Although, by definition, 10% of the total sample size would fall below this percentile, after observations had been classified according to the outcome variable, the actual numbers present for estimation in the sub-cells of the model could be relatively few, hence the large uncertainty in the estimates, especially in the small data sets. In addition, although not expected to have as high a risk of an adverse outcome as the 10th percentile group, those in the lowest quartile group should show a suffi- ciently elevated risk compared with the highest quartile, if the relationship between the indicator and the outcome is both of practical significance and monotonic with increasing values of the indicator. 18 WHO Bulletin OMS: Supplement Vol. 73 1995 Meta-analysis and preferred Indicators explanation of the inter-study variation in the ORs is sought by reference to the study characteristics described in Chapter 2, Table 6 (see pp. 12-14). These include study design (prospective versus retro- spective), prevalence level of the outcome, country groups, regional groups (Africa, southem Asia, south-east Asia, Latin America, USA and Europe), location of the study (urban, rural, mixed), decade of the study, WHO Collaborative study versus other, as well as matemal characteristics (mean matemal age, mean gestational age at delivery), etc. A number of these regressions are discussed in technical note B, Ch. 9, page 47. Interpretation. The follow-up adjustment for poten- tial biases by weighted regression analysis may often suggest an explanation-for example, under- lying group differences. These could perhaps be due to genuine differences in the strength of the relation- ship between the indicator and the outcome, depen- ding on the setting (e.g., the country, or character- istics of the mothers), or a reflection of the differences in study design and implementation, or perhaps both. When the basic hypothesis is that the strength of the underlying relationship between indi- cator X and outcome Y holds in all situations, observed differences need to be accounted for by first appealing to variation in the study charac- teristics before assuming that the differences are biological. Moreover, it should be remembered that even if the test for homogeneity is statistically signi- ficant, the inter-study differences in ORs may have little practical importance. This would certainly seem to be the case for many of the relationships studied. Which indicator? In the context of this study, the ideal indicator would be based on: - a simple single measurement (acceptable, low cost); - strong predictability or effect (high ORs); - reliable (narrow confidence interval for OR); - homogeneous (small inter-study variation); - timely (permitting effective intervention, includ- ing referral); - sensitive and specific for screening; - efficient in performance (low number of false classifications). It cannot be assumed therefore that the indica- tors with the highest OR values for a given outcome are the best as various other factors must be taken into consideration. For example, in Table 12 (Chap- ter 5) the highest single indicator OR for LBW relates to weight at the 9th lunar month (OR = 2.5); this is very late in pregnancy to initiate preventive action in the form of food supplementation. Conversely, pre- pregnancy weight and matemal height, while having lower OR values than weight at 9 months, are avail- able early enough to facilitate intervention. Consideration of relevant factors Assuming that an effective procedure is known and available with which to respond to a predicted poor pregnancy outcome, the selection of useful indicators of risk must be govemed by a number of additional considerations apart from a measure of predictive association, such as the odds ratio. These would include the sensitivity and specificity of the indica- tor, the prevalence of the condition being tested for, as well as the feasibility of obtaining the necessary measurement. Several authors have considered the issues conceming the selection of a 'best' indicator, particularly Galen & Gambino (13) and Habicht and co-workers (14-18). In this context, a critical ques- tion is which numerical quantity associated with an indicator is to be preferred? Should it be the highest odds ratio, or the highest sensitivity, or the highest specificity, or indeed the highest positive predictive value? Additional summary measures have been devised that combine a number of these qualities and might also be considered (13). High sensitivity is important in an indicator when the condition under consideration is serious, but treatable. For example, food and nutrient supple- mentation of the mother may be employed where a small-for-dates infants is anticipated; or there could be early referral of a mother to a central facility if pregnancy or delivery complications are considered likely. In this case a significant proportion of false positive results are felt to be acceptable because the treatment is not harmful to the patient and the system can cope with the extra service load. On the other hand, a test with high specificity should be considered if the outcome could be serious but not necessarily curable and where false positive diagnoses are distressing for the patient, e.g., when testing for congenital or genetic abnormalities. Positive predictive value is an important test characteristic if the treatment of a false positive may result in serious side-effects, such as in some can- cers. One additional summary measure might be con- sidered: test efficiency. A test with high efficiency minimizes the numbers of both false positives and false negatives; this would be important when a con- dition is serious and treatment carries some risk and WHO Bulletin OMS: Supplement Vol. 73 1995 19 Chapter 4 both false positives and false negatives have equally poor consequences. In practice, anthropometric assessment should be accompanied by a detailed obstetric history, phys- ical examination and, where available, diagnostic laboratory testing. Such additional information may be more critical than anthropometry in reaching a decision on risk status and case management. In effect, a combination of anthropometry and obstetric history will determine the joint sensitivity/specificity of the clinical examination. However, it will be recalled that the objective of this project concerns the relative performance of matemal anthropometric indicators alone. This will be especially relevant to those settings in which decisions are currently being made by a minimally trained health worker, particu- larly in small rural health posts or at community or home levels. Finally, the scope of anthropometry to predict adverse outcomes for the fetus and the mother is lim- ited to those conditions principally determined by maternal nutritional status, both past and current, including fetal and infant growth and difficult or obstructed labour. Other non-nutritional factors including age, parity, obstetrical history and ante- natal care have important implications for both the fetus and the mother and are strongly associated with both infant and maternal mortality. The role of sensitivity and specificity We have stated that considerations other than effect size (predictive power) have a bearing on the choice of a preferred indicator for field application. These are principally the sensitivity (Se) and specificity (Sp) of the indicator, its positive predictive value (PPV), and, of course, the feasibility of the measure- ment(s) in the particular setting. How do Se, Sp and PPV relate to the odds ratios? Can we expect the indicators with high ORs to do well in terms of these measures? These questions cannot be directly ad- dressed on the basis of the original analysis format for relative risk, i.e., comparison of lowest versus highest quartiles, which implies two cut-off points (the 25th and 75th percentiles). However, indicator sensitivity and specificity are concerned with rates of misclassification and must be based on a single cut- off point as individual mothers will be screened posi- tive or negative in relation to this point alone. Accor- dingly, all analyses were rerun using the 25th percentile cut-off point and the necessary measures were derived.b Therefore in the discussion that fol- lows, mention of sensitivity, specificity, positive pre- dictive value and odds ratio is now with reference to this analysis format. ORs are more strongly related to the sum of sen- sitivity and specificity rather than to either measure alone (exemplified in technical note D in Ch. 9, page 50); and positive predictive value is high when Se + Sp is high and declines when that sum falls. As will be seen below, supplementing information on one indicator with a second, e.g., attained weight coupled with knowledge of pre-pregnancy weight or matemal height, generally leads to a notable increase in the OR, and this is associated with higher indicator sen- sitivity, but usually at the expense of reduced speci- ficity. So although higher ORs may result, this does not automatically produce a preferred indicator. Ideally, a useful indicator will combine a statistically significant odds ratio with good specificity and sen- sitivity. Candidate lists for 'best' indicators We are still left with the question of which measure is to be preferred when choosing an indicator of risk for a particular outcome. The 1980 paper by Habicht (17) provides the most useful guidance in our situation. He states that when "various indicators of nutritional status are being compared [in order] to choose the best... screening test, ... that one which has the highest sensitivity at high levels of specifici- ty, even if the sensitivity is lower than that of another indicator at lower specificity", should be chosen. This suggestion can be readily applied to our results. For the anthropometric indicators, specificity rarely exceeds 0.9 and is typically found in the range 0.65 to 0.8 while sensitivity ranges from 0.25 to 0.45. Se and Sp values for each indicator and outcome combi- nation from all studies were collated and if for LBW and IUGR a given indicator's Sp was >0.7 and its Se was >0.35, then that indicator was marked out. For the remaining outcomes a slightly less stringent cri- terion was established: Sp>0.7 and Se >0.3. The results of the exercise were summarized and the proportion of cases in which an indicator met the Se/Sp criterion was calculated. A simple perform- ance measure was decided upon: if an indicator met the above criterion in over 40% of the data sets for which it was available, it was considered a candidate for 'best' indicator. The results are presented for each outcome in the relevant section and summarized in Chapter 8. b Sensitivity and specificity were estimated using the methodol- ogy of Coughlin et al. (19). 20 WHO Bulletin OMS: Supplement Vol. 73 1995