1	USING META-ANALYTIC TECHNIQUES TO MEASURE PSYCHOLOGICAL CHANGE IN PRIMARY STUDIES
2	Aim To explain how effect sizes and confidence intervals can be employed in primary studies as indicators of amount of psychological change.
3	Everything Depends On efficacy of the intervention validity, sensitivity and relevance of the DV appropriateness of the analyses interpretation of the results -> understanding and controlling of causative processes
4	Outline The Significance Testing Controversy What is Meta-analysis? Effect Sizes Interpretation of Effect sizes Confidence Intervals Graphical Displays Benchmarking & Comparisons Future directions
5	The Significance Testing Controversy Statistical significance testing was developed by Fisher to determine whether some agricultural techniques were superior to other techniques
6	Which is More Impressive? Statistical significance in a study with: N=10? N=100?
7	"Statistical significant testing has been..." Statistical significant testing has been utilised with little adaptation in psychological research, even though quite different questions are often being asked This has undermined the value of much psychological research
8	"Calls for a shift away..." Calls for a shift away from significance testing have been largely unheeded for approx. 30 years
9	The Four Possible Outcomes of a Significance Test
10	Power Power ~.60 in social science research i.e. on average, 40% chance of Type II error Under reporting of power Under reporting of effect sizes
11	"“Despite numerous efforts to..." “Despite numerous efforts to change self-concept there appears to be no consistent answer as to whether it is possible” - Janet Hattie (1992, p.221)
12	"Ways of Measuring Psychological Change" Ways of Measuring Psychological Change Clinical Observation/Opinion Difference Scores T Scores Significance Testing Effect Sizes & Confidence Intervals
13	"Ways of Reviewing Research on..." Ways of Reviewing Research on Psychological Change Traditional Literature Review Vote Counting Secondary Analysis Meta-analysis Mega-analysis
14	"Psychotherapy Debate" Psychotherapy Debate To counter what appeared to be selectivity of studies included in a review of psychotherapy effects by Eysenck, Glass introduced a procedure he termed meta-analysis. [1976,1977]
15	What is Meta-analysis? Equivalent to traditional (qualitative) review paper Enters summary quantitative data from each study into a new database, with IV codings Overall effects are summarised and variance predicted Used in medicine, psychology and education Outcome measure of interest is the ‘effect size’
16	Effect Sizes A standardised measure of ‘how much change’ OR ‘how much shared variation’
17	Effect Sizes Cohen’s d Hedge’s g Pearson’s r ANOVA - eta-square, omega-squared Regression - R squared Categorical - Phi & Cramer’s V
18	Cohen’s d - norms - control group - pooled
19	Cohen’s d A measure of the difference between two means in standard deviation units. d is equivalent to the differences between two z scores
20	Interpreting Effect Size (d) -ve = negative change 0 = no change +ve = positive change
21	Interpreting Effect Size (d) Cohen (1977): .2 = small .5 = moderate .8 = large Wolf (1986): .25 = educationally significant .50 = practically signficant (therapeutic) ESs are proportional, e.g., .40 is twice as much change as .20
22	Interpreting Effect Size (d) No agreed standards Interpretation is subjective Best approach - compare with previous findings
23	Comparative Effect Sizes (d) Adult psychotherapy outcomes : .68 (Smith, Glass & Miller, 1980) Children psychotherapy outcomes : .71 (Casey & Berman, 1980) Classroom intervention - Achievement : .40 (cited in Hattie, Marsh, Neill, & Richards, 1997) Classroom intervention - Affective : .28 (cited in Hattie, Marsh, Neill, & Richards, 1997) Self-concept intervention programs : .37 (Hattie, J.A., 1992)
24
25	OE Effect Sizes (d) Adolescent OE programs (43 studies) : .31 (Cason & Gillis, 1994) All OE research (96 studies) : .34 (cited in Hattie, Marsh, Neil, & Richards, 1997) Adventure Therapy - LOC : .38 (Hans, 1997, 2000) USA summer camps with self-focus : .41 (cited in Hattie, Marsh, Neil, & Richards, 1997)
26
27	Interpretation of Outdoor Education Effect Sizes (d)
28	Other ES Interpretations Psychotherapy 30% improvement for average client Classroom-based affective programs 11% improvement for average students Outdoor education 13% improvement for average participant 65% of OE participants are better off than people who don’t do an OE program (35% are not better off!)
29	Confidence Intervals
30	Graphing ESs & Confidence Intervals
31	Everything Still Depends On efficacy of the intervention validity, sensitivity and relevance of the DV appropriateness of the analyses interpretation of the results -> understanding and controlling causative processes
32	Looking Into The Future Benchmarking for program evaluation and quality assurance Increasing opportunities for cumulative, primary data research MA may become common expectation for literature reviewing
33	Conclusions Use MAs and ESs in your literature reviews Report ESs and CIs for your primary data Discuss relevant ES comparisons Suggest benchmarks When reporting significance, report power
34	References Abbott, C. (1987). Does outdoor education really work? Perks, research and reality. Journal of Adventure Education and Outdoor Leadership, 4(2), 22-25. Caldarella, P., & Merrell,-K.W. (1997). Common dimensions of social skills of children and adolescents: A taxonomy of positive behaviors. School Psychology Review, 26, 264-278. Cason D., & Gillis, H.L. (1994). A meta-analysis of outdoor adventure programming with adolescents. Journal of Experiential Education, 17(1), 40-47. Cohen, J. (1977). Statistical power analysis for behavioral sciences (revised ed.). New York: Academic Press. Hans, T. (1997). A meta-analysis of the effects of adventure programming on locus of control. Unpublished Master of Science thesis, Psychology Graduate Faculty, Georgia College, Milledgeville, GA. Hans, T. (2000). A meta-analysis of the effects of adventure programming on locus of control. Journal of Contemporary Psychotherapy, 30(1),33-60.
35	References Hattie, J.A. (1992). Enhancing self-concept. In J.M. Hattie. Self-concept (pp.221-240). New York: Lawrence Erlbaum. Hattie, J. (1992). Measuring the effects of schooling. Australian Journal of Education, 36(1), 5-13. Hattie, J. (1992). Self-concept. New York:: Lawrence Erlbaum. Hattie, J., Marsh, H.W., Neill, J.T. & Richards, G.E. (1997). Adventure Education and Outward Bound: Out-of-class experiences that have a lasting effect. Review of Educational Research, 67, 43-87. Lawson, M. (1997, November 24). Wilderness training yet to prove its worth.. The Australian Financial Review (p.7). Neill, J.T., & Richards, G.E. (1998). Does outdoor education really work? A summary of recent meta-analyses. Australian Journal of Outdoor Education, 3(1), 2-9. Marsh, P.E. (2000).* Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy. Baltimore: Johns Hopkins University Press. Wolf, F.M. (1986). Meta-analysis: Quantitative Methods for Research Synthesis. Beverly Hills, CA: Sage.