Course Business l New datasets on CourseWeb for Week 5 l But first, - - PowerPoint PPT Presentation

Course Business

l New datasets on CourseWeb for Week 5

l But first, we will finish last week’s math.csv

l Midterm assignment: Review a journal article in

your area that uses mixed-effects models

l Goals:

l Practice interpreting mixed-effects models l Springboard for class discussion of current practice &

reporting

l See CourseWeb document for specific requirements l Due on CourseWeb on October 24th at 1:30 PM– 4

weeks from today

l Grading rubric on CourseWeb

l New datasets on CourseWeb for Week 5

l But first, we will finish last week’s math.csv

l Midterm assignment: Review a journal article in

your area that uses mixed-effects models

l Requirements on the chosen article:

l Journal article, not poster / conference proceedings l Should have at least one random effect—that makes it a

mixed-effects model

l Any type of random effects structure OK (nested or

crossed)

l Random effect(s) can be anything—classrooms, schools,

subjects, items, families, …

l Can run the article by me if unsure

Course Business

Distributed Practice!

• Alicia is studying the effects of paternal involvement in child
• development. She samples 400 families, and then, within each

family, often includes multiple children. In this case, the level-2 model would describe ______ and the level-1 model would describe _______.

• Jason is studying autobiographical memory (memory about life

events). He samples 40 members of the Pittsburgh community as research participants. Jason asks each research participant to think of 15 important memories from their life and to rate various properties of each memory. (The 40 subjects all have different memories.) He is interested in whether features such as vividness and recency predict confidence in the memory. In this random effects structure, ______ are nested within _______.

Distributed Practice!

• Alicia is studying the effects of paternal involvement in child
• development. She samples 400 families, and then, within each

family, often includes multiple children. In this case, the level-2 model would describe families and the level-1 model would describe children.

• Jason is studying autobiographical memory (memory about life

events). He samples 40 members of the Pittsburgh community as research participants. Jason asks each research participant to think of 15 important memories from their life and to rate various properties of each memory. (The 40 subjects all have different memories.) He is interested in whether features such as vividness and recency predict confidence in the memory. In this random effects structure, memories are nested within participants.

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

Recap: Nested Random Effects

Student 1 Student 2 Student 3 Student 4

Level-1 model: Sampled STUDENTS

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Level-2 model: Sampled CLASSROOMS

• Level-2 model is for the superordinate level here,

Level-1 model is for the subordinate level

Recap: Nested Random Effects

• We included Classroom as a random effect in

the model

• model1 <- lmer(FinalMathScore ~ 1 + TOI +

(1|Classroom), data=math) Additional, unexplained subject variance (even after accounting for classroom differences) Variance of classroom intercepts (normal distribution with mean 0)

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

Level-2 Variables

• So far, all our model says about classrooms is

that they’re different

• Some classrooms have a large intercept
• Some classrooms have a small intercept
• But, we might also have some interesting

variables that characterize classrooms

• They might even be our main research interest!
• How about teacher theories of intelligence?
• Might affect how they interact with & teach students

Level-2 Variables

Student 1 Student 2 Student 3 Student 4

Sampled STUDENTS

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Sampled CLASSROOMS

• TeacherTheory characterizes Level 2
• All students in the same classroom will have the

same TeacherTheory

• xtabs(~ Classroom + TeacherTheory, data=math)

LEVEL 2 LEVEL 1

TeacherTheory TOI

Level-2 Variables

• This becomes another variable in the level-2

model of classroom differences

• Tells us what we can expect this classroom to be like

Student Error

Ei(j)

=

End-of-year math exam score

+ +

Baseline

Yi(j) B00j

Growth mindset

γ100x1i(j)

U0j

=

Intercept

+

Overall intercept

B00j γ000

Teacher effect for this classroom (Error) LEVEL-1 MODEL (Student) LEVEL-2 MODEL (Classroom) Teacher mindset +

γ200x20j

Level-2 Variables

• Teacher mindset is a fixed-effect variable
• We ARE interested in the effects of teacher mindset
• n student math achievement … a research

question, not just something to control for

• Even if we ran this with a new random sample of 30

teachers, we WOULD hope to replicate whatever regression slope for teacher mindset we observe (whereas we wouldn’t get the same 30 teachers back)

Level-2 Variables

• Since R uses mixed effects notation, we don’t

have to do anything special to add a level-2 variable to the model

• model2 <- lmer(FinalMathScore ~ 1 + TOI

+ TeacherTheory + (1|Classroom), data=math)

• R automatically figures out TeacherTheory is a

level-2 variable because it’s invariant for each classroom

• We keep the random intercept for Classroom

because we don’t expect TeacherTheory will explain all of the classroom differences. Intercept captures residual differences.

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

Multiple Random Effects

• Hold on! Classrooms aren’t fully independent,
• either. Some of them are from the same school,

and some are from different schools.

School 1 School 2

Sampled CLASSROOMS Sampled STUDENTS

LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Multiple Random Effects

• Is SCHOOL a fixed effect or a random effect?
• These schools are just a sample of possible schools of

interest -> Random effect.

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Multiple Random Effects

• No problem to have more than 1 random effect in

the model! Try adding a random intercept for school.

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Multiple Random Effects

• model3 <- lmer(FinalMathScore ~ 1 + TOI

+ TeacherTheory + (1|Classroom) + (1|School), data=math)

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Multiple Random Effects

• This is an example of nested random effects.
• Each classroom is always in the same school.

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

New Dataset

• tutor.csv
• Effects of computer math tutor on FinalMathScores
• 25 students in each of 10 classrooms in each of 20

schools

• Once you get the dataframe loaded into R, you

might notice a problem. See if you can identify it and fix it. (Work with your table!)

New Dataset

• tutor.csv
• Effects of computer math tutor on FinalMathScores
• 25 students in each of 10 classrooms in each of 20

schools

• Once you get the dataframe loaded into R, you

might notice a problem. See if you can identify it and fix it. (Work with your table!)

• R thinks ClassroomID is a numeric variable
• tutor$ClassroomID <-

as.factor(tutor$ClassroomID)

Multiple Random Effects

• Let’s do an intervention: Hours of use of math

tutoring software

• Which level(s) of the model could this be at?

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Multiple Random Effects

• Let’s do an intervention: Hours of use of math

tutoring software

• Which level(s) of the model could this be at?

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

If use of the tutor characterizes a whole school

Multiple Random Effects

• Let’s do an intervention: Hours of use of math

tutoring software

• Which level(s) of the model could this be at?

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

If classrooms within a school vary in tutor use, but consistent within a classroom

Multiple Random Effects

• Let’s do an intervention: Hours of use of math

tutoring software

• Which level(s) of the model could this be at?

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

If students within a classroom varied in their tutor usage

Multiple Random Effects

• Can you find this out from R?

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

Multiple Random Effects

• Can you find this out from R?
• xtabs( ~ ClassroomID + TutorHours, data=tutor)

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

If classrooms within a school vary in tutor use, but consistent within a classroom

• Try running a model to look at the effect of

TutorHours on FinalMathScore

• Hint: Don’t forget to account for the clustering!
• Hint 2: You will need 2 random intercepts

Multiple Random Effects

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

If classrooms within a school vary in tutor use, but consistent within a classroom

• model.Intercepts <- lmer(FinalMathScore ~ 1 +

TutorHours + (1|ClassroomID) + (1|School), data=tutor)

Multiple Random Effects

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

If classrooms within a school vary in tutor use, but consistent within a classroom

• Which do you think is better?
• Teachers are allowed to choose how much they use

the tutor in their classroom

• We randomly assign a specific # of tutor hours

Multiple Random Effects

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

If classrooms within a school vary in tutor use, but consistent within a classroom

• Random assignment allows stronger claims about

causality

• But, model is implemented the same either way
• Just affects how we interpret the results

Multiple Random Effects

School 1 School 2

Sampled SCHOOLS Sampled CLASSROOMS Sampled STUDENTS

LEVEL 3 LEVEL 2 LEVEL 1 Student 1 Student 2 Student 3 Student 4

Mr. Wagner’s Class

Ms. Fulton’s Class Ms. Green’s Class Ms. Cornell’s Class

If classrooms within a school vary in tutor use, but consistent within a classroom

Introduction to Random Slopes

• Right now, our assumption is that schools differ

in their baseline math score (intercept)

• What other ways might schools differ?
• Some schools might use the tutor more effectively

than others

• Fidelity of implementation is a major concern in education

research!

Introduction to Random Slopes

• Overall relationship

between a classroom’s use of the tutor & their average math score

• Current assumption

is that the slope of this line (= tutor effect) is the same in every school

• 10

15 20 25 30 35 50 60 70 80 90 100 Hours that a classroom used the tutor Average final math score

Random Slopes

• That is, so far, our model

says that schools vary in baseline math score

• Random intercept
• And that every 1 hour of

tutor use ≈ 0.49 points gain in final math score

• Slope is a fixed effect

50 55 60 65 70 Hours that a classroom used the tutor Average final math score 5 10 Highland 50 55 60 65 70 Midland 50 55 60 65 70 Crescent

Different intercepts for each school Assumption is that slope is the same for all schools

Random Slopes

• That is, so far, our model

says that schools vary in baseline math score

• Random intercept
• And that every 1 hour of

tutor use ≈ 0.49 points gain in final math score

• Slope is a fixed effect
• Schools may also vary

in how much tutor affects their classrooms’ scores

• A random slope of tutor

usage by schools

• Such differences may correlate with baseline diff.s

1 2 3 500 600 700 800 900 1000 1100 1200 Word frequency RT Subject 1 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 2 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 3 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 4 50 55 60 65 70 Hours that a classroom used the tutor Average final math score 5 10 Highland 50 55 60 65 70 Midland 50 55 60 65 70 Crescent

Schools still vary in intercept Slopes now also differ across sampled schools Schools that start off worse show a bigger benefit of tutor

Random Slopes: Statistical Consequences

• Important to capture this variability in the model
• Observations from the same school will show more

similar effects of tutor use

• Different from what other schools would be like
• If we don’t account

for this similarity, non-independence that underestimates true variability

• Inflates false positive rate

Highland School (slope 0.82) Classroom 21, 27 hrs Highland School (slope 0.82) Classroom 22, 28 hrs Highland School (slope 0.82) Classroom 29, 25 hrs Highland School (slope 0.82) Classroom 30, 21 hrs Midland School (slope 0.17) Classroom 97, 27 hrs Midland School (slope 0.17) Classroom 98, 28 hrs

t =

Estimate

• Std. error

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

Notation

• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours

γ100x1j(k)

U0j(k)

=

Classroom intercept

+

β0jk γ000

Teacher effect for this classroom (Error) Overall baseline

+

Baseline for students in this classroom

Notation

• Level-3 model of school k:
• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours

γ100x1j(k)

U0j(k)

=

Classroom intercept

+

β0jk δ00k

Teacher effect for this classroom (Error) Baseline for classrooms in this school

+

Baseline for students in this classroom School effect for this school (Error)

V00k

=

School intercept

+

δ00k γ000

Overall baseline

Notation

• Level-3 model of school k:
• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours

γ100x1j(k)

U0j(k)

=

Classroom intercept

+

β0jk δ00k

Teacher effect for this classroom (Error) Baseline for classrooms in this school

+

Baseline for students in this classroom School effect for this school (Error)

V00k

=

School intercept

+

δ00k γ000

Overall baseline

Right now, tutor slope γ100 is still just a fixed value. Maybe, like the intercept, we should allow to vary across schools.

• Level-3 model of school k:
• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours

δ10kx1j(k)

U0j(k)

=

Classroom intercept

+

β0jk δ00k

Teacher effect for this classroom (Error) Baseline for classrooms in this school

+

Baseline for students in this classroom School effect for this school (Error)

V00k

=

School intercept

+

δ00k γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10k

=

Tutor slope

+

δ10k γ100

Overall tutor slope (fixed effect)

• Level-3 model of school k:
• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours

δ10kx1j(k)

U0j(k)

=

Classroom intercept

+

β0jk δ00k

Teacher effect for this classroom (Error) Baseline for classrooms in this school

+

Baseline for students in this classroom School effect for this school (Error)

V00k

=

School intercept

+

δ00k γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10k

=

Tutor slope

+

δ10k γ100

Overall tutor slope (fixed effect)

Let’s start doing our algebraic substitution

• Level-3 model of school k:
• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours

δ10kx1j(k)

U0j(k)

=

Classroom intercept

+

β0jk δ00k

Teacher effect for this classroom (Error) Baseline for classrooms in this school

+

Baseline for students in this classroom School effect for this school (Error)

V00k

=

School intercept

+

δ00k γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10k

=

Tutor slope

+

δ10k γ100

Overall tutor slope (fixed effect)

Let’s start doing our algebraic substitution

• Level-3 model of school k:
• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours

δ10kx1j(k)

U0j(k)

=

Classroom intercept

+

β0jk

Teacher effect for this classroom (Error)

+

Baseline for students in this classroom By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10k

=

Tutor slope

+

δ10k γ100

Overall tutor slope (fixed effect)

Let’s start doing our algebraic substitution

• Level-3 model of school k:
• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours

δ10kx1j(k)

U0j(k)

=

Classroom intercept

+

β0jk

Teacher effect for this classroom (Error)

+

Baseline for students in this classroom By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10k

=

Tutor slope

+

δ10k γ100

Overall tutor slope (fixed effect)

Let’s start doing our algebraic substitution

• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours effect (overall & by school)

(γ100 + V10k)x1j(k)

U0j(k)

=

Classroom intercept

+

β0jk

Teacher effect for this classroom (Error)

+

Baseline for students in this classroom By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline

Let’s start doing our algebraic substitution

• Level-2 model of classroom j:
• Level 1 model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk) β0jk

Tutor hours effect (overall & by school)

(γ100 + V10k)x1j(k)

U0j(k)

=

Classroom intercept

+

β0jk

Teacher effect for this classroom (Error)

+

Baseline for students in this classroom By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline

Let’s start doing our algebraic substitution

• Mixed-effects model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk)

Tutor hours effect (overall & by school)

(γ100 + V10k)x1j(k)

U0j(k)

+

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline

• Mixed-effects model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk)

Tutor hours effect (overall & by school)

(γ100 + V10k)x1j(k)

U0j(k)

+

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline

Apply distributive property

• Mixed-effects model of student i:

Student Error

Ei(jk)

=

End-of-year math exam score

+

Yi(jk)

Fixed effect of tutor hours (overall)

γ100x1j(k) U0j(k)

+

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10kx1j(k)

+

FIXED RANDOM

• Because we have more than one random variable,

they can also covary

Covariance Matrix

[ ]

σ2u00j0 σ2u10j0 cov(σ2u00j0, u10j0) cov(σ2u00j0, u10j0)

Variance of subject intercept Variance of TutorHours slope across schools The correlation parameter!

Covariance matrix for subject random effects in maximal model

[ ]

σ2u00j0 σ2u10j0

In near- maximal model without the correlation parameter

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

Random Slopes: Implementation

=

End-of-year math exam score

Yi(jk)

Fixed effect of tutor hours (overall)

γ100x1j(k) U0j(k)

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10kx1j(k)

+

FIXED RANDOM

• So, lmer model:
• model.Slope <- lmer(FinalMathScore ~

Next, we need overall (fixed) effects of Intercept and TutorHours

+

Random Slopes: Implementation

=

End-of-year math exam score

Yi(jk)

Fixed effect of tutor hours (overall)

γ100x1j(k) U0j(k)

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10kx1j(k)

+

FIXED RANDOM

• So, lmer model:
• model.Slope <- lmer(FinalMathScore ~ 1 +

TutorHours Next, we need overall (fixed) effects of Intercept and TutorHours

+

Random Slopes: Implementation

=

End-of-year math exam score

Yi(jk)

Fixed effect of tutor hours (overall)

γ100x1j(k) U0j(k)

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10kx1j(k)

+

FIXED RANDOM

• So, lmer model:
• model.Slope <- lmer(FinalMathScore ~ 1 +

TutorHours How about a random difference in the intercept for each ClassroomID?

+

Random Slopes: Implementation

=

End-of-year math exam score

Yi(jk)

Fixed effect of tutor hours (overall)

γ100x1j(k) U0j(k)

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10kx1j(k)

+

FIXED RANDOM

• So, lmer model:
• model.Slope <- lmer(FinalMathScore ~ 1 +

TutorHours + (1|ClassroomID) How about a random difference in the intercept for each ClassroomID?

+

Random Slopes: Implementation

=

End-of-year math exam score

Yi(jk)

Fixed effect of tutor hours (overall)

γ100x1j(k) U0j(k)

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10kx1j(k)

+

FIXED RANDOM

• So, lmer model:
• model.Slope <- lmer(FinalMathScore ~ 1 +

TutorHours + (1|ClassroomID) Lastly, both intercept and TutorHours slope need to vary across schools

+

Random Slopes: Implementation

=

End-of-year math exam score

Yi(jk)

Fixed effect of tutor hours (overall)

γ100x1j(k) U0j(k)

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10kx1j(k)

+

FIXED RANDOM

• So, lmer model:
• model.Slope <- lmer(FinalMathScore ~ 1 +

TutorHours + (1|ClassroomID) + (1+TutorHours|School) Lastly, both intercept and TutorHours slope need to vary across schools

+

Random Slopes: Implementation

=

End-of-year math exam score

Yi(jk)

Fixed effect of tutor hours (overall)

γ100x1j(k) U0j(k)

Teacher effect for this classroom (Error)

+

By-school adjustment for intercept (Error)

V00k

+

γ000

Overall baseline By-school adjustment for tutor slope (Error)

V10kx1j(k)

+

FIXED RANDOM

• So, lmer model:
• model.Slope <- lmer(FinalMathScore ~ 1 +

TutorHours + (1|ClassroomID) + (1+TutorHours|School), data=tutor)

+

Random Slopes: Implementation

• Here’s our final model again:
• model.Slope <- lmer(FinalMathScore ~

1 + TutorHours + (1|ClassroomID) + (1 + TutorHours|School), data=tutor)

Like a miniature model formula for things we think will vary by schools Schools differ in their intercept (baseline math score) Schools differ in the effectiveness of the tutor

• n their math scores

Random Slopes: Output

Still have a fixed effect of TutorHours— slope estimated across all subjects & classrooms NEW: How much schools vary from the mean TutorHours slope

The fixed effect is reliable. In this case, we can conclude that final math scores are generally higher for classrooms that used the tutor more, even if there are some differences across schools in the size

• f this effect.

Correlation: Schools with a lower starting score show a larger benefit of the tutor

Random Slopes: Model Comparison

• Now that we’ve properly accounted for clustering, our estimate of the

standard error of the tutor effect is higher … and t value lower

• Excluding the slope would have increased our Type I error rate

As compared to the intercept-only model, our estimate of baseline school variance has now decreased—many of these unexplained “baseline” differences have now been explained as differing effectiveness of the tutor NEW MODEL (WITH SLOPE) OLD MODEL (INTERCEPT ONLY)

Random Effects: Implementation

• By the way, how come we never discussed a

(1+TutorHours|Classroom) slope?

• Within a school, we

can calculate a regression line relating tutor use to average math score

• Each class = 1 point
• 20

22 24 26 28 30 32 65 70 75 80 85 Hours that a classroom used the tutor Average final math score Class 30 Class 23 Class 29 Class 26 Class 27 Class 21 Class 22 Class 24 Class 25 Class 28

Random Effects: Implementation

• By the way, how come we never discussed a

(1+TutorHours|Classroom) slope?

• But each class has
• nly a single value of

TutorHours

• Classroom 22 used it

for 28 hours

• No way to draw a line

relating different values

• f TutorHours to score
• Need 2 points for a line!
• 20

22 24 26 28 30 32 65 70 75 80 85 Hours that a classroom used the tutor Average final math score Class 22

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

Testing Random Effects

• Does this random slope contribute significantly?
• i.e., is there significant variation in the effectiveness
• f the tutor across schools?
• We can compare the fit of models with &

without the random slope

• Using the likelihood-ratio test
• Same as when we fixed effects last week

Testing Random Effects

• Does this random slope contribute significantly?
• i.e., is there significant variation in the effectiveness
• f the tutor across schools?
• We can compare the fit of models with &

without the random slope

• Using the likelihood-ratio test
• Same as when we tested fixed effects last week
• anova(model.Intercepts, model.Slope)
• Could test random intercepts the same way

Model with slope fits significantly better!

Testing Random Effects

• Remember that this tests whether there is

significant variation in the slope across schools

• i.e., more than expected under H0 of no variation
• More complex model will always fit numerically

the same or better

• But if three schools had slopes of 0.39, 0.38,

and 0.40, this would probably not be significant

• Consistent with what’s expected from sampling

error

Testing Random Effects

• Caveats:
• Overfitting / shrinkage
• Problem we talked about last week

about using the data to decide the

• model. “Using up” degrees of freedom
• A new sample might yield a different random effects

structure

• In cases where we have a clear sampling

design, would want model to reflect that

• e.g., weird not to include random intercepts of

classroom & school here

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

An Experimental Dataset

• naming.csv
• RT to name/read a word aloud for students learning

English

• 60 Subjects each presented with 49 words (Items)

we randomly picked out of a dictionary

• Each row is a single trial (one subject responding to
• ne word)
• We’re interested in YearsOfStudy (of English) and

WordFreq (word frequency)

• Also interested in their interaction
• Which of these should we consider Fixed

Effects? Which are Random Effects?

An Experimental Dataset

• naming.csv
• RT to name/read a word aloud for students learning

English

• 60 Subjects each presented with 49 words (Items)

we randomly picked out of a dictionary

• Each row is a single trial (one subject responding to
• ne word)
• We’re interested in YearsOfStudy (of English) and

WordFreq (word frequency)

• Also interested in their interaction
• Which of these should we consider Fixed

Effects? Which are Random Effects?

• Fixed: YearsOfStudy, WordFreq, & interaction
• Random: Subject, Item

Crossed Random Effects

• Now, we have >1 trial per Subject
• Level-1 observations (RTs) are nested within subjects
• Clear we need to take account of this
• Some subjects will have faster RTs in general than others

Subjects (Level-2)

Trials (Level-1) RT 1 RT 2 RT 3 RT 4

Subject 1 Subject 2

Crossed Random Effects

• Try an R command to look at the categories of

the Item variable. Do some seem easier than

• thers?

Crossed Random Effects

• Try an R command to look at the categories of

the Item variable. Do some seem easier than

• thers?
• summary(naming$Item)

Probably relatively easy Probably relatively hard!

Crossed Random Effects

• Each word is also used in more than 1 trial
• Want to take account of that, too
• Observations from the same item will be more similar to
• ne another, too (“boy” easier than “carburetor”)

Subjects (Level-2)

Trials (Level-1) RT 1 RT 2 RT 3 RT 4

Subject 1 Subject 2

Words

“Boy” “Carbu retor”

Crossed Random Effects

• In fact, we can think of each trial as the pairing of a

subject and a word

Subjects (Level-2)

Trials (Level-1) RT 1 RT 2 RT 3 RT 4

Subject 1 Subject 2

Words (Level-2)

“Boy” “Carbu retor”

Crossed Random Effects

• Random effects not hierarchically nested
• Before: Each classroom appears in only 1 school
• Here: Each item presented to each subject
• If we draw all the lines in, we see they cross

Subjects (Level-2)

Trials (Level-1) RT 1 RT 2 RT 3 RT 4

Subject 1 Subject 2

Words (Level-2)

“Boy” “Carbu retor”

Crossed Random Effects

• Crossed random effects structure
• a/k/a cross-classified when we’re dealing with

existing classifications

Subjects (Level-2)

Trials (Level-1) RT 1 RT 2 RT 3 RT 4

Subject 1 Subject 2

Words (Level-2)

“Boy” “Carbu retor”

Crossed Random Effects

• Conceptually different sampling, but same syntax!
• Try fitting a model with:
• Fixed effects of YearsOfStudy, WordFreq, & interaction
• Random intercepts for Subject and Item

Subjects (Level-2)

Trials (Level-1) RT 1 RT 2 RT 3 RT 4

Subject 1 Subject 2

Words (Level-2)

“Boy” “Carbu retor”

Crossed Random Effects

• model1 <- lmer(RT ~ 1+ YearsOfStudy * WordFreq +

(1|Subject) + (1|Item), data=naming) 2 significant main effects, but no interaction Greater Subject variance than Item variance Typical in experiments because we

• ften design

items to be similar

Crossed Random Effects

• Huge improvement over ANOVA analyses in

psycholinguistics / experimental psychology!

• We used to have to conduct separate analyses to

assess generalizability over subjects & items

• Or, item effects were simply ignored!

Subjects (Level-2)

Trials (Level-1) RT 1 RT 2 RT 3 RT 4

Subject 1 Subject 2

Words (Level-2)

“Boy” “Carbu retor”

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

• What kind of random slopes might be relevant

here?

• To answer this question, we need to understand

how “between-subjects variables” differ from “within-subjects variables”

Between vs Within Subjects

Between vs Within Subjects

• In an experimental design, some variables are:

Ø Between-Subjects variables: Each subject is in 1 and only 1 group … or has 1 and only 1 value

• Randomly assigned to drug 1 vs. drug 2 vs. placebo
• Demographic variables; e.g., SES
• Cognitive/linguistic differences (e.g., working mem. score)
• “Between subjects” because differences in this variable

are only seen between one subject and another SUBJECT 10’s DATA Native speaker Trial 1: Correct Trial 2: Correct Trial 3: Incorrect SUBJECT 11’s DATA Non-native speaker Trial 1: Incorrect Trial 2: Correct Trial 3: Incorrect

Between vs Within Subjects

• In an experimental design, some variables are:

Ø Between-Subjects variables: Each subject is in 1 and only 1 group … or has 1 and only 1 value

• Randomly assigned to drug 1 vs. drug 2 vs. placebo
• Demographic variables; e.g., SES
• Cognitive/linguistic differences (e.g., working mem. score)
• “Between subjects” because differences in this variable

are only seen between one subject and another

Ø Within-Subjects variables: Same subject sees more than 1 condition or has >1 value

• Same subject sees both congruent (green) and

incongruent (blue) Stroop trials

• Values that vary w/in a study, e.g., # of previous trials
• Variables where you’d use a repeated measures ANOVA
• “Within-subjects” because you can see differences in this

variable even within a single subject

Between vs Within Subjects

• In an experimental design, some variables are:

Ø Between-Subjects variables: Each subject is in 1 and only 1 group … or has 1 and only 1 value

• Randomly assigned to @
• Demographic variables; e.g., SES
• Cognitive/linguistic differences (e.g., working mem. score)
• “Between subjects” because differences in this variable

are only seen between one subject and another

Ø Within-Subjects variables: Same subject sees more than 1 condition or has >1 value

• Same subject sees both congruent (green) and

incongruent (blue) Stroop trials

• Values that vary w/in a study, e.g., # of previous trials
• Variables where you’d use a repeated measures ANOVA
• “Within-subjects” because you can see differences in this

variable even within a single subject SUBJECT 12’S DATA Trial 1: Congruent Stroop, 655 ms Trial 2: Incongruent Stroop, 512 ms Trial 3: Incongruent Stroop, 711 ms Trial 4: Congruent Stroop: 642 ms

• The same variable could end up as between- or

within-subjects, depending on experimental design

• I’m interested in maintenance rehearsal (repetition)
• vs. elaborative rehearsal (relating to other concepts)
• Half of my participants study second-language

vocab using maintenance rehearsal, and half study words using elaborative rehearsal à Between Subjects

• Each participant studies some words with

maintenance rehearsal and some with elaborative rehearsal à Within Subjects

Between vs Within: Advanced

• I assign 40 students to study for an upcoming

biology exam by practicing retrieving the facts (retrieval practice) and 40 students to re-read the textbook (restudy). Practice Type is…

• In a visual perception task, 20 subjects try to

determine as quickly as possible whether a circle is present on the screen (among other objects). Each subject sees ½ of the trials where the square is present and ½ where it’s not. Trial Type is…

• Kevin compares 100 first-generation college

students to 100 college students who are not first- generation students in their feeling of belonging at their university. College history is…

Between vs Within: Practice

• I assign 40 students to study for an upcoming

biology exam by practicing retrieving the facts (retrieval practice) and 40 students to re-read the textbook (restudy). Practice Type is…

• Between subjects
• In a visual perception task, 20 subjects try to

determine as quickly as possible whether a circle is present on the screen (among other objects). Each subject sees ½ of the trials where the square is present and ½ where it’s not. Trial Type is…

• Within subjects
• Kevin compares 100 first-generation college

students to 100 college students who are not first- generation students in their feeling of belonging at their university. College history is…

• Between subjects

Between vs Within: Practice

• I recruit 40 Psych Subject Pool students for my
• experiment. Each participant reads 20 syntactically

ambiguous sentences and 20 unambiguous

• sentences. Sentence type is…
• I recruit 40 Psych Subject Pool students for my
• experiment. 20 participants read syntactically

ambiguous sentences, and the other 20 read only unambiguous sentences. Sentence type is…

Between vs Within: More Practice

• I recruit 40 Psych Subject Pool students for my
• experiment. Each participant reads 20 syntactically

ambiguous sentences and 20 unambiguous

• sentences. Sentence type is…
• Within subjects
• I recruit 40 Psych Subject Pool students for my
• experiment. 20 participants read syntactically

ambiguous sentences, and the other 20 read only unambiguous sentences. Sentence type is…

• Between subjects

Between vs Within: More Practice

• Each of 100 participants in my study makes a

decision in each of 2 different moral dilemmas: one involving direct harm, and one involving indirect

• harm. I also measure participants’ working memory

and split them into “low WM” and “high WM” groups.

• Moral dilemma type is…
• Working memory is…

Between vs Within: More Practice

• Each of 100 participants in my study makes a

decision in each of 2 different moral dilemmas: one involving direct harm, and one involving indirect

• harm. I also measure participants’ working memory

and split them into “low WM” and “high WM” groups.

• Moral dilemma type is…
• Within subjects
• Working memory is…
• Between subjects

Between vs Within: More Practice

• How about in our naming.csv dataset?
• Word frequency is…
• Each subject sees high-, medium- and low-frequency

words

• Within subjects
• Years of study is…
• This experiment takes place at a single point in time,

so the number of years a subject has been studying English is fixed and never varies

• Between subjects

Between vs Within: More Practice

When are Random Slopes Appropriate?

• If the random effect is subject…
• Random slopes for subjects appropriate for:
• Within-subjects variables
• We can draw a regression line for each subject
• Calculate the effect “within” each subject
• Random slopes for subjects inappropriate for:
• Between-subjects variables
• Can’t draw a regression line within each subject

Subject 1 High Frequency Subject 1 Low Frequency Subject 2 High Frequency Subject 2 Low Frequency Subject 1: 6 years of study Subject 2: 2 years of study

Random Slopes: Implementation

• Remember, original model was:
• model1 <- lmer(RT ~ 1 + WordFreq * YearsOfStudy

+ (1|Subject) + (1|Item), data=naming)

• What variable varies within subjects? Try adding a

random slope for it

Random Slopes: Implementation

• Remember, original model was:
• model1 <- lmer(RT ~ 1 + WordFreq * YearsOfStudy

+ (1|Subject) + (1|Item), data=naming)

• With slope:
• model2 <- lmer(RT ~ 1 + WordFreq * YearsOfStudy

+ (1+WordFreq|Subject) + (1|Item), data=naming) Again, miniature model formula for things we think will vary by subjects Subjects differ in their intercept (baseline RT) Subjects differ in the effect of word frequency

• n their RTs

Random Slopes

• Original model says

that subjects vary in baseline RT

• Random intercept
• And that a 1-unit change

in word frequency ≈ 120 ms decrease in RT

• Slope is a fixed effect

1 2 3 500 600 700 800 900 1000 1100 1200 Word frequency RT Subject 1 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 2 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 3 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 4

Different intercepts for each subject Slope is the same for everyone

Random Slopes

• Original model says

that subjects vary in baseline RT

• Random intercept
• And that a 1-unit change

in word frequency ≈ 120 ms decrease in RT

• Slope is a fixed effect
• Slope captures how

ppl vary in their sensitivity to word frequency

• Random slope
• Such differences may correlate with baseline diff.s

1 2 3 500 600 700 800 900 1000 1100 1200 Word frequency RT Subject 1 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 2 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 3 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 4 1 2 3 500 600 700 800 900 1000 1100 1200 Word frequency RT Subject 1 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 2 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 3 1 2 3 500 600 700 800 900 1000 1100 1200 Subject 4

Subjects still vary in intercept Slopes now also differ across sampled subjects Subjects with higher baselines also have steeper slopes

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

Between vs Within Items

• We can draw a similar distinction between

Ø Between-Items variables: Each item appears in 1 and only 1 condition … or has 1 and only 1 value

• Visual complexity of pictures
• One set of sentences is used in our “plausible” condition

and a completely different set is used in our “implausible” condition

• “Between items” because differences in this variable are
• nly seen between one item and another

Ø Within-Items variables: Same item appears in more than 1 condition or has >1 value

• Same science facts presented (across subjects) in either

a elaborative- or maintenance-rehearsal condition

• We manipulate the same fictitious resume to have either

a stereotypically African-American or European-American name

• I present the same 40 visual search displays to 10

people with ADHD and 10 people without ADHD. ADHD status is…

• For his psycholinguistic experiment, Goofus writes

10 grammatical sentences and 10 completely different ungrammatical sentences. Grammaticality is…

• Gallant, for his psycholinguistic experiment, writes

10 sentences, and then tweaks the verb in each sentence to create both a grammatical and an ungrammatical version. Grammaticality is…

Between vs Within Items

• I present the same 40 visual search displays to 10

people with ADHD and 10 people without ADHD. ADHD status is…

• Between items
• For his psycholinguistic experiment, Goofus writes

10 grammatical sentences and 10 completely different ungrammatical sentences. Grammaticality is…

• Between items
• Gallant, for his psycholinguistic experiment, writes

10 sentences, and then tweaks the verb in each sentence to create both a grammatical and an ungrammatical version. Grammaticality is…

• Within items

Between vs Within Items

Between vs Within Items

• Try updating the most recent model with

random slope(s) for the within-items variable(s)

• Hint: Think about whether each variable is constant

for a particular word or not

• The final full model:
• model3 <- lmer(RT ~

1 + YearsOfStudy * WordFreq + (1 + WordFreq|Subject) + (1 + YearsOfStudy|Item), data=naming)

Between vs Within Items

• Why didn’t we include a

random slope of WordFreq by items?

• Each word has a constant

frequency (within this experiment)

• Doesn’t make sense to discuss effect
• f word frequency within an item
• No random slope of frequency by

items

• But, each item IS presented to

different subjects who differ in their

YearsOfStudy

1 2 3 4 300 400 500 600 700 Word frequency RT

computer

1 2 3 4 300 400 500 600 700

panther

1 2 3 4 300 400 500 600 700

verify

1 2 3 4 300 400 500 600 700

memorandum

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

When Are Random Slopes Necessary?

• Remember that failing to account for clustering

could inflate our Type I error rate

• With factorial experiments, ideal to include all

random slopes we can (Barr et al., 2013)

• Maximal random effects structure
• Expectation is that we assume that subjects could

vary in, say, their WordFreq effect—that’s why we ran more than one subject

• Possible (even likely) that this may not converge.

We’ll discuss this next week.

When Are Random Slopes Necessary?

• Remember that failing to account for clustering

could inflate our Type I error rate

• In more observational data, not necessarily the

case you’d include all possible random slopes

• What’s theoretically relevant or expected?

Week 5: Random Slopes

l Finish Nested Random Effects

l Recap l Level-2 Variables l Multiple Random Effects

l Random Slopes

l Introduction l Notation l Implementation l Testing Random Effects

l Crossed Random Effects

l Examples l Random Slopes, Revisited

l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?

The View Ahead

• We’ve covered the basics of fitting a mixed

effects model

• But only with continuous variables
• Next 3 weeks: Categorical variables (factors)
• Next two weeks: Categorical predictors (IVs)
• e.g., experimental vs. control condition; race/ethnicity
• After that: Categorical outcomes (DVs)
• e.g., recalled vs. didn’t recall a science fact; did or didn’t

graduate high school