Course Business l Two new datasets for class today: l CourseWeb: - - PowerPoint PPT Presentation
Course Business l Two new datasets for class today: l CourseWeb: Course Documents Sample Data Week 4 l Another relevant package: apaTables l Next two weeks: Random effects for different types of designs l This week: Nested random
l Two new datasets for class today:
l CourseWeb: Course Documents à Sample Data
l Another relevant package: apaTables l Next two weeks: Random effects for different
l This week: “Nested” random effects l Next week: “Crossed” random effects
Kuznetsova, Brockhoff, & Christensen, 2017
l How are degrees of freedom estimated?
StutteringSeverity + (1|Subject) + (1|Item), data=stuttering)
StutteringSeverity + (1|Subject) + (1|Item), data=stuttering)
(1|Subject), data=cog.aging)
Age:PhysicalActivity + (1|Subject), data=cog.aging)
StutteringSeverity + (1|Subject) + (1|Item), data=stuttering)
StutteringSeverity + (1|Subject) + (1|Item), data=stuttering)
severity and quality of life (in addition to the main effects)
(1|Subject), data=cog.aging)
Age:PhysicalActivity + (1|Subject), data=cog.aging)
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
l Social support & health (e.g., Cohen & Wills, 1985)
l lifeexpectancy.csv: l Longitudinal study of 1000 subjects – some
l Perceived social support (z-scored) l Lifespan l And several control variables
l Last week, we saw you could fit several
l model1 <- lmer(RT ~ 1 + PrevTrials +
FontSize + (1|Subject) + (1|Item), data=Stroop)
l model2 <- lmer(RT ~ 1 + PrevTrials +
FontSize + PrevTrials:FontSize + (1|Subject) + (1|Item), data=Stroop)
l Or:
l my.model <- lmer(Lifespan ~ 1 + SocSupport +
YrsEducation + (1|Family), data=lifeexpectancy)
l your.model <- lmer(Lifespan ~ 1 +
HrsExercise + Conscientiousness + (1|Family), data=lifeexpectancy)
Ø Theoretical Model #1: Social support does affect health Ø Theoretical Model #2: Social support does not affect health
l Three possible models of life expectancy:
l Amount of weekly exercise l Amount of weekly exercise & perceived social
l Amount of weekly exercise, perceived social
l These are nested models—each one can be
l Three possible models of life expectancy:
l Amount of weekly exercise l Amount of weekly exercise & perceived social
l Amount of weekly exercise, perceived social
l Which set of information would give us the
l Three possible models of life expectancy:
l Amount of weekly exercise l Amount of weekly exercise & perceived social
l Amount of weekly exercise, perceived social
Slope of regression line relating last name vowels to life expectancy is near 0 But that merely fails to improve predictions; doesn’t hurt them
l We can compare nested models (only) using
l Remember that likelihood is what we search for in
l We can compare nested models (only) using
l First, fit each of the models to be compared:
l Try fitting a model1 that includes both HrsExercise
l Then, a model2 that omits SocSupport
(1) = 8.67, p < .01
Log likelihood will also be somewhat higher (better) for the complex model … but is it SIGNIFICANTLY better? We’ll discuss what this means in a moment (don’t worry; it’s what we want)
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
l Let’s think about our two models: l What are some possible values of γ200 (the
l 3.83 l -1.04 l 0 – there is no social support effect
model1
model2
l Let’s think about our two models: l What happens when γ200 is equal to 0?
l Anything multiplied by 0 is 0, so SocSupport just
l Becomes the same thing as model2
model1
model2
l Let’s think about our two models: l model2 is just a special case of model1
l The version of model1 where γ200 = 0 l One of many possible versions of model1 l Why we say model2 is “nested” in model1
model1
model2
l Let’s think about our two models: l This also helps show why model1 always
l model1 can account for the case where γ200 = 0 l But it can also account for many other cases, too
model1
model2
l Let’s think about our two models:
l Testing whether model2 fits significantly better is the
l i.e., whether there is a significant effect of SocSupport
l LR test is another way of doing hypothesis testing!
model1
model2
l Let’s think about our two models:
l “But you’re just comparing two models! You’re not
l Closely related to our research goal: Which
l The theoretical model where social support doesn’t affect
life expectancy
l The model where social support does affect life expectancy
model1
model2
p-value from likelihood ratio test: .0032 p-value from lmerTest t- test: .0033
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
SentenceType:Aphasia
SentenceType:Aphasia
model1
model2
Yang, 2005; Oehlert, 2012
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
Paper 1 Length: 6 pages Models run: 2 Paper 2 Length: 5 pages Models run: 1 Paper 3 Length: 3 pages Models run: 4 DESERVED SCORE DARTBOARD OF SAMPLING ERROR! RESULTING GRADE
Paper 1 Length: 6 pages Models run: 2 Paper 2 Length: 5 pages Models run: 1 Paper 3 Length: 3 pages Models run: 4 DESERVED SCORE DARTBOARD OF SAMPLING ERROR! RESULTING GRADE
(Babyak, 2004, p. 414)
recessions since the end of WW II
strong correlation somewhere, even if it’s just sampling error
significant) weather stations & stocks we searched through
true on a new set of weather data & stock prices
Vul et al., 2009
excludes error that doesn’t
personality in a new, independent dataset
“Just as you do not need to know exactly how a car engine works in order to drive safely, you do not need to understand all the intricacies of the economy to accurately read those gauges.” – Economic forecasting firm ECRI (quoted in Silver, 2012)
“There is really nothing so practical as a good theory.”
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
l For each item, rate your agreement on a scale
DEFINITELY AGREE DEFINITELY DISAGREE
DEFINITELY AGREE DEFINITELY DISAGREE
DEFINITELY AGREE DEFINITELY DISAGREE
DEFINITELY AGREE DEFINITELY DISAGREE
l Subtract your total from 21, then divide by 3 l Learners hold different views of intelligence
FIXED MINDSET: Intelligence is fixed. Performance = ability GROWTH MINDSET: Intelligence is malleable Performance = effort
End-of-year math exam score
Growth mindset
End-of-year math exam score
Baseline
Growth mindset
Error
End-of-year math exam score
Baseline
Growth mindset
Student 1 Student 2 Student 3 Student 4
Sampled STUDENTS
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
Sampled CLASSROOMS Math achievement score y11 Theory of intelligence score x111 Independent error term e11 Math achievement score y21 Theory of intelligence score x121 Independent error term e21 Math achievement score y42 Theory of intelligence score x142 Independent error term e42
Student 1 Student 2 Student 3 Student 4
Sampled STUDENTS
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
Sampled CLASSROOMS Math achievement score y11 Theory of intelligence score x111 Independent error term e11 Math achievement score y21 Theory of intelligence score x121 Independent error term e21 Math achievement score y42 Theory of intelligence score x142 Independent error term e42
size, teaching style, teacher’s experience…
But if we have a lot of kids from the same classroom, they share more similarities than all kids in population Understating the standard error across subjects… …thus overstating the significance test
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
(1|Classroom), data=math)
(1|Classroom), data=math)
(1|Classroom), data=math)
(1|Classroom), data=math)
Additional, unexplained subject variance (even after accounting for classroom differences) Variance of classroom intercepts (normal distribution with mean 0)
Variance of Random Effect of Interest Sum of All Random Effect Variances
Classroom Variance Classroom Variance + Residual Variance
classroom = Classroom matters a lot = high ICC
classroom = Classroom not that important = low ICC
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
Student Error
End-of-year math exam score
Baseline
Growth mindset
Student Error
End-of-year math exam score
Baseline
Growth mindset
Student Error
End-of-year math exam score
Baseline
Growth mindset
Intercept
Overall intercept across everyone
Teacher effect for this classroom (Error)
Student Error
End-of-year math exam score
Baseline
Growth mindset
Intercept
Teacher effect for this classroom (Error) LEVEL-1 MODEL (Student) LEVEL-2 MODEL (Classroom) Overall intercept across everyone
Student 1 Student 2 Student 3 Student 4
Level-1 model: Sampled STUDENTS
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
Level-2 model: Sampled CLASSROOMS
Student Error
End-of-year math exam score
Baseline
Growth mindset
Intercept
Teacher effect for this classroom (Error) LEVEL-1 MODEL (Student) LEVEL-2 MODEL (Classroom) Overall intercept across everyone
Student Error
End-of-year math exam score
Growth mindset
Overall intercept
Teacher effect for this classroom (Error)
Student Error
End-of-year math exam score
Growth mindset
Overall intercept
Teacher effect for this class (Error)
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
Student 1 Student 2 Student 3 Student 4
Sampled STUDENTS
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
Sampled CLASSROOMS
LEVEL 2 LEVEL 1
TeacherTheory TOI
Student Error
End-of-year math exam score
Baseline
Growth mindset
Intercept
Overall intercept
Teacher effect for this classroom (Error) LEVEL-1 MODEL (Student) LEVEL-2 MODEL (Classroom) Teacher mindset +
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
School 1 School 2
Sampled CLASSROOMS Sampled STUDENTS
LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
If use of the tutor characterizes a whole school
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
If classrooms within a school vary in tutor use, but consistent within a classroom
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
If students within a classroom varied in their tutor usage
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
If classrooms within a school vary in tutor use, but consistent within a classroom
TeacherTheory + (1|Classroom) + (1|School), data=math)
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
If classrooms within a school vary in tutor use, but consistent within a classroom
l Model Comparison
l Nested Models
l Hypothesis Testing l REML vs ML
l Non-Nested Models l Shrinkage
l Nested Random Effects
l Introduction to Clustering l Random Effects l Modeling Random Effects l Notation l Level-2 Variables l Multiple Random Effects l Limitations & Future Directions
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
School 1 School 2
Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS
LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4
Mr. Wagner’s Class
Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class
Subjects
Reading Times RT 1 RT 2 RT 3 RT 4
Subject 1 Subject 2
Sentences
Item A Item B
Subjects
Reading Times RT 1 RT 2 RT 3 RT 4
Subject 1 Subject 2
Sentences
Item A Item B