Course Business l Sign up for final project presentations l Dec 5 th - - PowerPoint PPT Presentation
Course Business l Sign up for final project presentations l Dec 5 th : Doug, Jenny, Kelly, Kole, Rob, Zac l Dec 12 th : Ciara, Griff, Kori, Lauren, Lin, Rebecca l Shorter lecture on effect size & power l sleep.csv on CourseWeb l Scott to be out
l Sign up for final project presentations
l Dec 5th: Doug, Jenny, Kelly, Kole, Rob, Zac l Dec 12th: Ciara, Griff, Kori, Lauren, Lin, Rebecca
l Shorter lecture on effect size & power
l sleep.csv on CourseWeb
l Scott to be out of town the rest of the week
l E-mail responses may be slow l Slides posted next week
l NO CLASS NEXT WEEK
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
l In each comparison, delete only observations if
l Correlating Extraversion & Conscientiousness
l In each comparison, delete only observations if
l Correlating Extraversion & ReadingSpan à
l Avoids data loss l But, results not completely consistent /
l e.g., possible to have A > B > C > A
l
cor.test(stress$ReadingSpan, stress$Extraversion)
l
cor.test(stress$Conscientiousness, stress$Extraversion)
d.f.s don’t match because they’re based on different subsets of the data
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
l Delete any observation where data is missing
l e.g., stress2 <- na.omit(stress)
l Default in lmer() and many other software
l Avoids inconsistency l In some cases, could result in a lot of data loss
l However, mixed effects models do well even with
l Unlike ANOVA, MEMs properly account for some
l Avoids inconsistency l In some cases, could result in a lot of data loss
l However, mixed effects models do well even with
l Unlike ANOVA, MEMs properly account for some
l Produces the correct parameter estimates if
l Although some other things (R2) may be incorrect
l Estimates will be wrong if missingness is non-
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
l Replace missing values with the mean of the
l Imputing the mean reduces the variance
l This increases chance of detecting spurious effects
l Also distorts the correlations with other variables l Bad. Don’t do this!
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
l Replace missing values with the values
l
interimModel <- lmer(ReadingSpan ~ 1 + OperationSpan + (1|Subject), data=stress)
l
Create a model that predicts RSpan from OSpan
l
predictedValues <- predict(interimModel, stress[is.na(stress$ReadingSpan),]
l
Use the model to predict the missing ReadingSpan values
l
stress[is.na(stress$ReadingSpan),'ReadingSpan'] <- predictedValues
l Replace the missing ReadingSpan values with the predicted
Reading Span
Reading Span = NA NA Operation Span
l Replace missing values with the values
l If ignorable missingness, get the correct
l And, standard errors not as distorted
l Especially if we add some noise to the fitted values
l predictedValues <- predictedValues +
Reading Span
Reading Span = NA NA Operation Span
l Replace missing values with the values
l Where this is useful?
l Many observations have a small
l Listwise deletion would wipe out
Reading Span
Reading Span = NA NA Operation Span
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
l Like doing conditional imputation several times
l Replace missing data with one possible set of values l Run the model l Repeat it
l Final result averages these
Dataset with imputation 1 Dataset with imputation 2 Dataset with imputation 3 Model results 1 Model results 2 Model results 3 Final Results
Dataset with missing data {
l R package mice (have to install)
l Schematic example:
l imp <- mice(stress) l Creates several sets of imputed data l Need to set some parameters to indicate level 1 vs level 2
l miModel <- with(imp, lmer(…) )
l Fit the model to each set of imputed data
l result <- pool(miModel)
l Combine the model results
l summary(result) l Limitations:
l Limited to two nested levels (with current software) l Only gives you fixed effect estimates (not estimates of
l Can be time-consuming
l Alternative: Pattern-mixture models
l Classify participants by the patterns of missing data l Then look at the effects / pattern of results within
l e.g., Effects of reading span for people with personality data
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
LARGE EFFECT SIZE, LOW RELIABILITY
SMALL EFFECT SIZE, HIGH RELIABILITY
October 26, 2015
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day POOR recall of RACCOON GOOD recall of RACCOON
Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day TEST A: Study RACCOON 5 sec. Study WITCH 5 sec. Study RACCOON 5 sec. 10 sec 10 sec 1 day TEST B:
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
6 8 10 2 4 6 8 10 12 PREDICTED hours of sleep ACTUAL hours of sleep
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
1 2 3
Total probability
here under H0 = .05
Total probability
here under H0 = .05
Total probability
here under H0 = .05
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
p-value happens to be higher in this slightly larger sample
Now, p-value happens to be lower
SIGNIFICANT!! PUBLISH NOW!!
But not significant in this even larger sample
YES
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
Collaboration, 2015)
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
Set population parameters Mean = 723 ms Group difference = 100 ms Subject var = 30
Set population parameters Mean = 723 ms Group difference = 100 ms Subject var = 30 Create a random sample from these data N subjects = 20 N items = 40
Set population parameters Mean = 723 ms Group difference = 100 ms Subject var = 30 Create a random sample from these data N subjects = 20 N items = 40 Run our planned model and see if we get a significant result
Set population parameters Mean = 723 ms Group difference = 100 ms Subject var = 30 Create a random sample from these data N subjects = 20 N items = 40 Run our planned model and see if we get a significant result Repeat with a new sample from the same population
Set population parameters Mean = 723 ms Group difference = 100 ms Subject var = 30 Create a random sample from these data N subjects = 20 N items = 40 Run our planned model and see if we get a significant result Repeat with a new sample from the same population
Set population parameters Mean = 723 ms Group difference = 100 ms Subject var = 30 Create a random sample from these data N subjects = 60 N items = 40 Run our planned model and see if we get a significant result Repeat with a new sample from the same population
Set population parameters Mean = 723 ms Group difference = 100 ms Subject var = 30 Create a random sample from these data N subjects = 60 N items = 40 Run our planned model and see if we get a significant result Repeat with a new sample from the same population
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power
l Missing Data Solutions
l Casewise Deletion l Listwise Deletion l Unconditional Imputation l Conditional Imputation l Multiple Imputation
l Effect Size
l Unstandardized l Standardized l Interpreting Effect Size l Variance Explained
l Power
l Recap of Null Hypothesis Significance Testing l Why Should We Care? l Estimating Effect Size l Doing Your Own Power Analysis l Influences on Power