Meta-Analysis

views updated

META-ANALYSIS

Meta-analysis is the quantitative review of the results of a number of individual studies in order to integrate their findings. The term (from the Greek meta meaning after) refers to analysis of the conclusions of the original analyses. The methodology can in principle be applied to quantitative studies in any area of investigation, but it has become a basic tool in healthcare research. It is part of the broader approach of research synthesis, which also includes qualitative aspects.

Evolution of Meta-Analysis

Gaining an overview of the outcomes of different experiments is the constant aim of science, and statisticians have been concerned with the combination of results since the emergence of formal statistical inference in the early twentieth century. The basic principles were established by the 1950s (Cochran 1954), and the need became clear with the subsequent rapid increase in research publications. The procedure was first developed in the social sciences, and the term meta-analysis introduced in the educational literature in 1976. The 1980s saw mounting interest in the combination of results of clinical trials, and since the early 1990s meta-analysis has experienced explosive growth in medical applications.

Although there seems little doubt that meta-analysis is here to stay, it has been fraught with controversy. There is the problem of the quality of individual studies, with their own biases, often small clinical trials with poor design and execution. There is the problem of heterogeneity, studies that measured different effects, used different populations, had different aims. A further problem is that of publication bias, the fact that studies with positive results are more likely to get published than those with negative outcomes, leading to an inflation of the effect estimate. Related to this is Tower of Babel bias, meaning that most meta-analyses identify only reports published in English.

An international conference on meta-analysis was held in Germany in 1994, to review problems and progress (Spitzer 1995). A strong opponent present called the method "statistical alchemy for the 21st century" (Feinstein 1995). But work has continued, with the development of guidelines for doing meta-analyses, emphasizing the need to identify unpublished studies, eliminate incomplete reports and those of flawed research designs, and include only quality studies that appear to address the same well-defined question. The gold standard is that of Individual Patient Data (IPD), where the original data are available for reanalysis in the combined context. Cumulative meta-analysis is the systematic updating of the analysis as new results become available. There is also extensive research on meta-analysis for observational studies.

The Cochrane Collaboration

An important, promising development is the vigorous Cochrane Collaboration, "an international nonprofit and independent organization, dedicated to making up-to-date, accurate information about the effects of health care readily available worldwide. It produces and disseminates systematic reviews of health care interventions and promotes the search for evidence in the form of clinical trials and other studies of interventions" (Cochrane Collaboration). The movement was inspired by Archibald Cochrane (1909–1988), the British epidemiologist best known for his 1972 work Effectiveness and Efficiency: Random Reflections on Health Services. Cochrane urged equitable provision of those modes of healthcare that had been shown effective in properly designed studies, preferably randomized clinical trials. He considered the latter among the most ethical forms of treatment, and he emphasized the need for systematic critical summaries, with periodic update by specialty, of all relevant randomized clinical trials.

The first Cochrane Center opened in the United Kingdom in 1992, followed by the founding of the Cochrane Collaboration in 1993. In November 2004 its web site listed twelve Cochrane centers worldwide (using six languages) that serve as reference centers for 192 nations and coordinate the work of thousands of investigators. The main output of the Cochrane Collaboration is the Cochrane Library (CLIB), published and updated quarterly by Wiley InterScience and available by subscription via the Internet and on CD-ROM. Its contents include the Cochrane Database of Systematic Reviews (CDSRs), over 3,000 reviews prepared by fifty Collaborative Review Groups (CRGs), the Cochrane Central Register of Controlled Trials, bibliographic data on hundreds of thousands of controlled trials, as well as methodologic information on the rapidly developing field of research synthesis, and critical assessment of systematic reviews carried out by others.

The Ethics of Evidence

Meta-analysis, an attempt to integrate the information already on hand from past studies, enhanced by guidelines that it be done on the highest professional level, fits into the framework of the Ethics of Evidence, a multidisciplinary approach proposed for dealing with the uncertainties of medicine (Miké 1999). The Ethics of Evidence calls for the development, dissemination, and use of the best possible evidence for decisions in healthcare. As a complementary precept, it points to the need to accept that there will always be uncertainty.

To explore the quality of evidence from meta-analyses, a 1997 study compared the results of twelve large randomized clinical trials (RCTs) published in four leading medical journals with the conclusions of nineteen previously published meta-analyses addressing the same questions, for a total of forty primary and secondary outcomes (LeLorier et al. 1997). The agreement between the meta-analyses and the subsequent large RCTs was only somewhat better than chance. A third of the meta-analyses failed to correctly predict the outcome of the RCTs, and would have led to adoption of an ineffective treatment or the rejection of a useful one. (The actual differences between effect estimates were not large, but that did not count in this adopt/reject type of analysis.) Then in 2002 the long-held belief that menopausal hormone replacement therapy offered protection against heart disease, a medical consensus supported by meta-analyses, was shockingly reversed by RCT evidence (Wenger 2003).

The Cochrane Collaboration, as a worldwide, integrated movement, has the great potential to promote cooperation on high-quality, controlled clinical trials. Systematic reviews of these, with regular update and dissemination, should help improve the evidence available for the practice of medicine. But it is important to keep in mind that even the best meta-analysis cannot take the place of original research. Evidence-based medicine, which makes heavy use of the results of meta-analyses, cannot apply evidence that does not exist. Scientists need to stay close to the primary literature, with an open mind, to get new ideas, seek new insights, and generate new hypotheses.

The public needs to have a cautious view of meta-analysis, judging each case in its proper context. For example, the meta-analysis showing that more than 100,000 Americans die each year from the side effects of legally prescribed drugs (Lazarou et al. 1998) merits serious concern, even if the estimate is not quite accurate. There is no substitute for being informed, getting involved, and taking personal responsibility.

VALERIE MIKÉ

BIBLIOGRAPHY

Bailar, John C., III. (1997). "The Promise and Problems of Meta-Analysis." New England Journal of Medicine 337: 559–560.

Cochran, William G. (1954). "The Combination of Estimates from Different Experiments." Biometrics 10: 101–129.

Cochrane, Archibald L. (1972). Effectiveness and Efficiency: Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust.

Feinstein, Alvan R. (1995). "Meta-Analysis: Statistical Alchemy for the 21st Century." Journal of Clinical Epidemiology 48(1): 71–86. Commentary by Alessandro Liberati, "A Plea for a More Balanced View of Meta-Analysis and Systematic Overviews of the Effect of Health Care Interventions," is included.

Lazarou, Jason; Bruce H. Pomeranz; and Paul N. Corey. (1998). "Incidence of Adverse Drug Reactions in Hospitalized Patients: A Meta-Analysis of Prospective Studies." Journal of the American Medical Association 279(15): 1200–1205.

LeLorier, Jacques; Geneviève Grégoire; Abdeltif Benhaddad; et al. (1997). "Discrepancies between Meta-Analyses and Subsequent Large Randomized, Controlled Trials." New England Journal of Medicine 337: 536–542.

Miké, Valerie. (1999). "Outcomes Research and the Quality of Health Care: The Beacon of an Ethics of Evidence." Evaluation & the Health Professions 22: 3–32. Commentary by Edmund D. Pellegrino, "The Ethical Use of Evidence in Biomedicine," also appears in this issue, pp. 33–43.

Moher, David. (2001). "QUOROM." [Quality of Reporting of Meta-Analyses] In Biostatistics in Clinical Trials, eds. Carol Redmond and Theodore Colton. New York: John Wiley & Sons.

Spitzer, Walter O., ed. (1995). The Potsdam International Consultation on Meta-Analysis, special issue Journal of Clinical Epidemiology 48(1): 1–172.

Stroup, Donna F., and Stephan B. Thacker. (2000). "Meta-Analysis in Epidemiology." In Encyclopedia of Epidemiologic Methods, eds. Mitchell H. Gail and Jacques Benichou. New York: John Wiley & Sons.

Thompson, Simon G. (2001). "Meta-Analysis." In Biostatistics in Clinical Trials, eds. Carol Redmond, and Theodore Colton. New York: John Wiley & Sons.

Wenger, Nanette K. (2003). "Menopausal Hormone Therapy and Cardiovascular Protection: State of the Data 2003." Journal of the American Medical Women's Association 58: 236–239.