Quantitative Research Methods

Meta-analysis
Research Methodology

James Neill
Last updated:
10 May 2006

Introduction

  • Meta-analysis is a statistical technique for amalgamating, summarising, and reviewing previous quantitative research.  By using meta-analysis, a wide variety of questions can be investigated, as long as a reasonable body of primary research studies exist.   Selected parts of the reported results of primary studies are entered into a database, and this "meta-data" is "meta-analyzed", in similar ways to working with other data - descriptively and then inferentially to test certain hypotheses.
  • Meta analysis can be used as a guide to answer the question 'does what we are doing make a difference to X?', even if 'X' has been measured using different instruments across a range of different people.  Meta-analysis provides a systematic overview of quantitative research which has examined a particular question.
  • The appeal of meta analysis is that it in effect combines all the research on one topic into one large study with many participants. The danger is that in amalgamating a large set of different studies the construct definitions can become imprecise and the results difficult to interpret meaningfully.
  • Not surprisingly, as with any research technique, meta-analysis has its advantages and disadvantages.  An advantage is its objectivity, and yet like any research, ultimately its value depends on making some qualitative-type contextualizations and understandings of the objective data. 
  • Meta-analysis has been used to give helpful insight into:
    • the overall effectiveness of interventions (e.g., psychotherapy, outdoor education),
    • the relative impact of independent variables (e.g., the effect of different types of therapy), and
    • the strength of relationship between variables.
  • To get more introduction to meta-analysis, go to

Effect Sizes & Confidence Intervals

  • Meta analysis reports findings in terms of effect sizes.  The effect size provides information about how much change is evident across all studies and for subsets of studies.
  • There are many different types of effect size, but they fall into two main types:
    • standardized mean difference (e.g., Cohen's d or Hedges g) or
    • correlation (e.g., Pearson's r)
  • It is possible to convert one effect size into another, so each really just offers a differently scaled measure of the strength of an effect or a relationship.
  • The standardised mean effect size is basically computed as the difference score divided by the standard deviation of the scores.
  • In meta-analysis, effect sizes should also be reported with:
    • the number of studies and the number of effects used to create the estimate.
    • confidence intervals to help readers determine the consistency and reliability of the mean estimated effect size.
  • For more information about calculating effect sizes and confidence intervals, see:
  • Tests of statistical significance can also be conducted and on the effect sizes.
  • Different effect sizes are calculated for different constructs of interest, as predetermined by the researchers based on what issues are of interest in the research literature.
  • Rules of thumb and comparisons with field-specific benchmarks can be used to interpret effect sizes.  According to an arbitrary but commonly used interpretation of effect size by Cohen (1988), a standardised mean effect size of 0 means no change, negative effect sizes mean a negative change, with .2 a small change, .5 a moderate change, and .8 a large charge. Wolf (1986), on the other hand, suggests that .25 is educationally significant and .50 is clinically significant.

Using Effect Sizes in Primary Studies

How to Conduct a Meta-analysis

Meta-analytic Studies of Psychological Interventions

Hattie, J. (1992). Self-concept. NJ: Lawrence Erlbaum.

Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment. American Psychologist, 48, 1181-1201.

Smith, M. L., Glass, G. V., & Miller, T. I. (1980).  The benefits of psychotherapy. Baltimore: Johns Hopkins University Press.

Meta-analysis Methodology References

Bushman, B. J., & Wells, G. L. (2001). Narrative impressions of literature: The availability bias and the corrective properties of meta-analytic approaches. Personality and Social Psychology Bulletin, 27, 1123-1130.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: Academic Press.

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.

Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351-379.

Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Beverly Hills, CA: Sage.