Invited Commentary: A Critical Look at Some Popular Meta-Analytic Methods

Abstract

Meta-analysis is essential for obtaining reproducible summaries of study results and valuable for discovering patterns among study results. A good meta-analysis will highlight and delineate the subjective components of these processes and vigorously search for sources of heterogeneity. Unfortunately, these objectives are not always met by common techniques. For example, a scatterplot is an objective summarization if the data are uncensored, but inferred patterns should be regarded as subjective recognitions of the analyst, not objective data properties. Random-effects summaries encourage averaging over important data patterns, divert attention from key sources of heterogeneity, and can amplify distortions produced by publication bias; such summaries should only be used when important heterogeneity remains after a thorough search for the sources of such heterogeneity. Quality scoring adds the analyst's subjective bias to the results, wastes information, and can prevent the recognition of key sources of heterogeneity; it should be completely replaced by meta-regression on quality items (the score components). Am J Epidemiol 1994; 140:290–6.