Back to 301

Tutorial 7:
Power, Effect Sizes, Confidence Intervals, and Sample Size

Last updated:
05 May 2005

Assumed knowledge

  • Basic Knowledge of properties of normal distribution, t-tests, ANOVA, MLR

Francis & Howell

  • Francis deals with power, effect sizes, confidence intervals and N within each type of analysis as opposed to having a stand alone section
  • This makes it hard to review in Francis, but my suggestion is to go through the pages listed in the Francis index for "effect sizes" and "power"
  • Howell is a little better, he has a chapter on Power (Ch15) but it doesn't really deal with effect sizes and confidence intervals, and he uses delta and gamma which he doesn't clearly link with commonly used effect sizes such as d, r, etc.- aargh!.  So, read through Ch15, but concentrate on the lecture/tutorial notes notes more so than with other topics so far.

Downloadable files

Power

  • Use power.xls

  • If I am using a confidence interval of  95% and I want a power level .8, what would be the minimum sample size to detect a standardised mean difference of .3?

  • If I am using a confidence interval of 99% and I want a power level of .7, what would be the minumum sample size to detect an effect size of .7?

  • If I am using a confidence interval of 90% and I want a power level of .9, with a sample size of 200, what effect size will I be able to detect?

  • If I am using a confidence interval of 95% and I will have a sample size of 50, and I want a power level of .8, what effect size will I be able to detect?

Effect Sizes & Confidence Intervals with Cohen's d

  • Use Cohensd.xls
  • What is the effect size between two means (and how large is it?), where M1 = 10, SD1 = 3, M2 = 12.5, SD2 = 4
    • Using a 99% confidence interval, N = 20 in each group, is this a significant difference?
    • Using a 95% confidence interval, N = 20 in each group, is this a significant difference?
    • Using a 90% confidence interval, N = 20 in each group, is this a significant difference?
    • What happens if we increase the N in each group to 100?
  • What is the effect size between two means (and how large is it?), where M1 = 8, SD1 = 2, M2 = 3.5, SD2 = 2
    • Using a 99% confidence interval, N = 40 in each group, is this a significant difference?
    • Using a 95% confidence interval, N = 40 in each group, is this a significant difference?
    • Using a 90% confidence interval, N = 40 in each group, is this a significant difference?
    • What happens if we decrease the N in each group to 10?
  • What is the effect size between two means (and how large is it?), where M1 = .5, SD 1= .1, M2 = .6, SD2 = .07
    • Using a 99% confidence interval, N = 200 in each group, is this a significant difference?
    • Using a 95% confidence interval, N = 200 in each group, is this a significant difference?
    • Using a 90% confidence interval, N = 200 in each group, is this a significant difference?
  • What is the effect size between two means, N = 15, where M1 = 6.5, SD1 = 1, M2 = 7.5, SD2 = 1.1
    • What is the confidence interval?  Using only, the confidence interval, is this difference significant?
    • How large would N need to be to get a just significant result?
    • Using N = 15, how large would M2 need to be to get a just significant result?
  • It is possible to work the % of difference between two means, by treating Cohen's d as a z-score in a normal distribution
    left bound right bound mean standard deviation

Eta-squared & R-squared

  • Use any dataset
  • Design an analysis that can be tackled using MLR and ANOVA
  • Obtain power (via the ANOVA analysis)
  • Compare eta-squared (from ANOVA) with R squared from MLR
  • Compare partial eta-squared (from ANOVA) with the beta coefficients from MLR
  • What can you conclude about the relationship between eta-squared and r-squared?

Error Bar Graphs

  • Use any dataset
  • Conduct a one-way ANOVA and graphically present the means and confidence intervals using an Error Bar Graph - is this error bar chart consistent with the statistical results?
  • Conduct a two-way ANOVA and graphically present the means and confidence intervals using an Error Bar Graph - is this error bar chart consistent with the statistical results?

Summary of Meta-analysis

  • Find a published meta-analysis
  • Summarise the meta-analysis in 250 words
  • Consider including:
    • The research question
    • The overall effect size and its size/significance - note which type of effect size is used, commonly r or d
    • Were there any predictors of the effect size, what were they, and how important were they?
    • Were there any other results of note (e.g, file drawer effect)?
    • What were the strengths, limitations and/or recommendations of the study?
  • Include APA citation
  • Note: There are no readings in the texts about meta-analysis, but there's plenty on the web - suggest google search or this starting point - Meta-analysis Research Methodology