USING META-ANALYTIC TECHNIQUES TO MEASURE PSYCHOLOGICAL CHANGE
IN PRIMARY STUDIES

Aim
To explain
how effect sizes and confidence intervals
can be employed
 in primary studies
as indicators of amount of psychological change.

Everything Depends On
efficacy of the intervention
validity, sensitivity and relevance of the DV
appropriateness of the analyses
interpretation of the results
-> understanding and controlling of causative processes

Outline
The Significance Testing Controversy
What is Meta-analysis?
Effect Sizes
Interpretation of Effect sizes
Confidence Intervals
Graphical Displays
Benchmarking & Comparisons
Future directions

The Significance Testing Controversy
Statistical significance testing was developed by Fisher to determine whether some agricultural techniques were superior to other techniques

Which is More Impressive?
Statistical significance in a study with:
 N=10?
 N=100?

"Statistical significant testing has been..."
Statistical significant testing has been utilised with little adaptation in psychological research, even though quite different questions are often being asked
This has undermined the value of much psychological research

"Calls for a shift away..."
Calls for a shift away from significance testing have been largely unheeded for approx. 30 years

The Four Possible Outcomes of a Significance Test

Power
Power ~.60 in social science research
i.e. on average, 40% chance of Type II error
Under reporting of power
Under reporting of effect sizes

"“Despite numerous efforts to..."
“Despite numerous efforts to change self-concept there appears to be no consistent answer as to whether it is possible”

- Janet Hattie (1992, p.221)

"Ways of Measuring Psychological Change"
Ways of Measuring Psychological Change

Clinical Observation/Opinion
Difference Scores
T Scores
Significance Testing
Effect Sizes & Confidence Intervals

"Ways of Reviewing Research on..."
Ways of Reviewing Research on Psychological Change

Traditional Literature Review
Vote Counting
Secondary Analysis
Meta-analysis
Mega-analysis

"Psychotherapy Debate"
Psychotherapy Debate
 To counter what appeared to be
selectivity of studies
included in a review of
psychotherapy effects by Eysenck,
 Glass introduced a procedure
he termed meta-analysis.
[1976,1977]

What is Meta-analysis?
Equivalent to traditional (qualitative) review paper
Enters summary quantitative data from each study into a new database, with IV codings
Overall effects are summarised and variance predicted
Used in medicine, psychology and education
Outcome measure of interest is the ‘effect size’

Effect Sizes
A standardised measure of
‘how much change’ OR
‘how much shared variation’

Effect Sizes
Cohen’s d
Hedge’s g
Pearson’s r
ANOVA - eta-square, omega-squared
Regression - R squared
Categorical - Phi & Cramer’s V

Cohen’s d
- norms
- control group
- pooled

Cohen’s d
A measure of
the difference between two means
 in standard deviation units.

d is equivalent to the differences between two z scores

Interpreting Effect Size (d)
-ve     = negative change
            0     = no change
          +ve   = positive change

Interpreting Effect Size (d)
Cohen (1977):   .2     = small
                                  .5     = moderate
                                  .8     = large
Wolf (1986):       .25     = educationally
      significant
                                  .50     = practically signficant
      (therapeutic)
ESs are proportional, e.g.,
.40 is twice as much change as .20

Interpreting Effect Size (d)
No agreed standards
Interpretation is subjective
Best approach
- compare with previous findings

Comparative Effect Sizes (d)
Adult psychotherapy outcomes              : .68
(Smith, Glass & Miller, 1980)
Children psychotherapy outcomes         : .71
(Casey & Berman, 1980)
Classroom intervention - Achievement  : .40
(cited in Hattie, Marsh, Neill, & Richards, 1997)
Classroom intervention - Affective         : .28
(cited in Hattie, Marsh, Neill, & Richards, 1997)
Self-concept intervention programs       : .37
(Hattie, J.A., 1992)

Slide 24

OE Effect Sizes (d)
Adolescent OE programs (43 studies)            : .31
(Cason & Gillis, 1994)
All OE research (96 studies)                       : .34
(cited in Hattie, Marsh, Neil, & Richards, 1997)
Adventure Therapy - LOC                      : .38
(Hans, 1997, 2000)
USA summer camps with self-focus      : .41
(cited in Hattie, Marsh, Neil, & Richards, 1997)

Slide 26

Interpretation of Outdoor Education Effect Sizes (d)

Other ES Interpretations
Psychotherapy
30% improvement for average client
Classroom-based affective programs
11% improvement for average students
Outdoor education
13% improvement for average participant
65% of OE participants are better off
than people who don’t do an OE program
(35% are not better off!)

Confidence Intervals

Graphing ESs & Confidence Intervals

Everything Still Depends On
efficacy of the intervention
validity, sensitivity and relevance of the DV
appropriateness of the analyses
interpretation of the results
-> understanding and controlling causative processes

Looking Into The Future
Benchmarking for program evaluation and quality assurance
Increasing opportunities for cumulative, primary data research
MA may become common expectation for literature reviewing

Conclusions
Use MAs and ESs in your literature reviews
Report ESs and CIs for your primary data
Discuss relevant ES comparisons
Suggest benchmarks
When reporting significance, report power

References
Abbott, C. (1987). Does outdoor education really work?  Perks, research and reality.  Journal of Adventure Education and Outdoor Leadership, 4(2), 22-25.
Caldarella, P., & Merrell,-K.W. (1997). Common dimensions of social skills of children and adolescents: A taxonomy of positive behaviors.  School Psychology Review, 26, 264-278.
Cason D., & Gillis, H.L. (1994). A meta-analysis of outdoor adventure programming with adolescents. Journal of Experiential Education, 17(1), 40-47.
Cohen, J. (1977). Statistical power analysis for behavioral sciences (revised ed.). New York: Academic Press.
Hans, T. (1997). A meta-analysis of the effects of adventure programming on locus of control. Unpublished Master of Science thesis, Psychology Graduate Faculty, Georgia College, Milledgeville, GA.
Hans, T. (2000).  A meta-analysis of the effects of adventure programming on locus of control.  Journal of Contemporary Psychotherapy, 30(1),33-60.

References
Hattie, J.A. (1992). Enhancing self-concept.  In J.M. Hattie. Self-concept (pp.221-240). New York: Lawrence Erlbaum.
Hattie, J. (1992). Measuring the effects of schooling.  Australian Journal of Education, 36(1), 5-13.
Hattie, J. (1992). Self-concept. New York:: Lawrence Erlbaum.
Hattie, J., Marsh, H.W., Neill, J.T. & Richards, G.E. (1997). Adventure Education and Outward Bound: Out-of-class experiences that have a lasting effect. Review of Educational Research, 67, 43-87.
Lawson, M. (1997, November 24). Wilderness training yet to prove its worth.. The Australian Financial Review (p.7).
Neill, J.T., & Richards, G.E. (1998). Does outdoor education really work?  A summary of recent meta-analyses.  Australian Journal of Outdoor Education, 3(1), 2-9.
Marsh, P.E. (2000).*
Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy. Baltimore: Johns Hopkins University Press.
Wolf, F.M. (1986). Meta-analysis: Quantitative Methods for Research Synthesis. Beverly Hills, CA: Sage.