USING METAANALYTIC
TECHNIQUES TO MEASURE PSYCHOLOGICAL CHANGE
IN PRIMARY STUDIES
To explain  
how effect sizes and confidence intervals  
can be employed  
in primary studies  
as indicators of amount of psychological change. 
efficacy of the intervention  
validity, sensitivity and relevance of the DV  
appropriateness of the analyses  
interpretation of the results  
> understanding and controlling of causative processes 
The Significance Testing Controversy  
What is Metaanalysis?  
Effect Sizes  
Interpretation of Effect sizes  
Confidence Intervals  
Graphical Displays  
Benchmarking & Comparisons  
Future directions 
The Significance Testing Controversy
Statistical significance testing was developed by Fisher to determine whether some agricultural techniques were superior to other techniques 
Statistical significance in a study with:  
N=10?  
N=100? 
"Statistical significant testing has been..."
Statistical significant testing has been utilised with little adaptation in psychological research, even though quite different questions are often being asked  
This has undermined the value of much psychological research 
Calls for a shift away from significance testing have been largely unheeded for approx. 30 years 
The Four Possible Outcomes of a Significance Test
Power ~.60 in social science research i.e. on average, 40% chance of Type II error 

Under reporting of power  
Under reporting of effect sizes 
"“Despite numerous efforts to..."
“Despite numerous efforts to change selfconcept there appears to be no consistent answer as to whether it is possible”  
 Janet Hattie (1992, p.221) 

"Ways of Measuring Psychological Change"
Ways of Measuring Psychological Change  
Clinical Observation/Opinion 

Difference Scores  
T Scores  
Significance Testing  
Effect Sizes & Confidence Intervals  
"Ways of Reviewing Research on..."
Ways of Reviewing Research on Psychological Change  
Traditional Literature Review 

Vote Counting  
Secondary Analysis  
Metaanalysis  
Megaanalysis  
Psychotherapy Debate  
To counter what appeared to be selectivity of studies included in a review of psychotherapy effects by Eysenck, Glass introduced a procedure he termed metaanalysis. [1976,1977] 

Equivalent to traditional (qualitative) review paper  
Enters summary quantitative data from each study into a new database, with IV codings  
Overall effects are summarised and variance predicted  
Used in medicine, psychology and education  
Outcome measure of interest is the ‘effect size’ 
A standardised measure of  
‘how much change’ OR  
‘how much shared variation’ 
Cohen’s d  
Hedge’s g  
Pearson’s r  
ANOVA  etasquare, omegasquared  
Regression  R squared  
Categorical  Phi & Cramer’s V  
 norms  
 control group  
 pooled 
A measure of the difference between two means in standard deviation units. 

d is equivalent to the differences between two z scores 
ve = negative change  
0 = no change  
+ve = positive change  
Cohen (1977): .2 = small  
.5 = moderate  
.8 = large  
Wolf (1986): .25
= educationally significant 

.50 = practically signficant (therapeutic) 

ESs are proportional, e.g., .40 is twice as much change as .20 
No agreed standards  
Interpretation is subjective  
Best approach  compare with previous findings 
Adult psychotherapy outcomes : .68 (Smith, Glass & Miller, 1980) 

Children psychotherapy outcomes : .71 (Casey & Berman, 1980) 

Classroom intervention 
Achievement : .40 (cited in Hattie, Marsh, Neill, & Richards, 1997) 

Classroom intervention  Affective : .28 (cited in Hattie, Marsh, Neill, & Richards, 1997) 

Selfconcept intervention programs : .37 (Hattie, J.A., 1992) 
Adolescent OE programs (43
studies) : .31 (Cason & Gillis, 1994) 

All OE research (96 studies) : .34 (cited in Hattie, Marsh, Neil, & Richards, 1997) 

Adventure Therapy  LOC : .38 (Hans, 1997, 2000) 

USA summer camps with selffocus : .41 (cited in Hattie, Marsh, Neil, & Richards, 1997) 
Interpretation of Outdoor Education Effect Sizes (d)
Psychotherapy 30% improvement for average client 

Classroombased affective programs 11% improvement for average students 

Outdoor education 13% improvement for average participant 

65% of OE participants are better
off than people who don’t do an OE program (35% are not better off!) 
Graphing ESs & Confidence Intervals
efficacy of the intervention  
validity, sensitivity and relevance of the DV  
appropriateness of the analyses  
interpretation of the results  
> understanding and controlling causative processes 
Benchmarking for program evaluation and quality assurance  
Increasing opportunities for cumulative, primary data research  
MA may become common expectation for literature reviewing 
Use MAs and ESs in your literature reviews  
Report ESs and CIs for your primary data  
Discuss relevant ES comparisons  
Suggest benchmarks  
When reporting significance, report power 
Abbott, C. (1987). Does outdoor education really work? Perks, research and reality. Journal of Adventure Education and Outdoor Leadership, 4(2), 2225.  
Caldarella, P., & Merrell,K.W. (1997). Common dimensions of social skills of children and adolescents: A taxonomy of positive behaviors. School Psychology Review, 26, 264278.  
Cason D., & Gillis, H.L. (1994). A metaanalysis of outdoor adventure programming with adolescents. Journal of Experiential Education, 17(1), 4047.  
Cohen, J. (1977). Statistical power analysis for behavioral sciences (revised ed.). New York: Academic Press.  
Hans, T. (1997). A metaanalysis of the effects of adventure programming on locus of control. Unpublished Master of Science thesis, Psychology Graduate Faculty, Georgia College, Milledgeville, GA.  
Hans, T. (2000). A metaanalysis of the effects of adventure programming on locus of control. Journal of Contemporary Psychotherapy, 30(1),3360.  
Hattie, J.A. (1992). Enhancing selfconcept. In J.M. Hattie. Selfconcept (pp.221240). New York: Lawrence Erlbaum.  
Hattie, J. (1992). Measuring the effects of schooling. Australian Journal of Education, 36(1), 513.  
Hattie, J. (1992). Selfconcept. New York:: Lawrence Erlbaum.  
Hattie, J., Marsh, H.W., Neill, J.T. & Richards, G.E. (1997). Adventure Education and Outward Bound: Outofclass experiences that have a lasting effect. Review of Educational Research, 67, 4387.  
Lawson, M. (1997, November 24). Wilderness training yet to prove its worth.. The Australian Financial Review (p.7).  
Neill, J.T., & Richards, G.E. (1998). Does outdoor education really work? A summary of recent metaanalyses. Australian Journal of Outdoor Education, 3(1), 29.  
Marsh, P.E. (2000).*  
Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy. Baltimore: Johns Hopkins University Press.  
Wolf, F.M. (1986). Metaanalysis: Quantitative Methods for Research Synthesis. Beverly Hills, CA: Sage.  