New Dataset • tutor.csv • Effects of computer math tutor on FinalMathScore s • 25 students in each of 10 classrooms in each of 20 schools • Once you get the dataframe loaded into R, you might notice a problem. See if you can identify it and fix it. (Work with your table!) • R thinks ClassroomID is a numeric variable tutor$ClassroomID <- • as.factor(tutor$ClassroomID)
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 • Let’s do an intervention: Hours of use of math tutoring software • Which level(s) of the model could this be at?
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 If use of the tutor characterizes a whole school LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 • Let’s do an intervention: Hours of use of math tutoring software • Which level(s) of the model could this be at?
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS If classrooms within a school vary in tutor use, but consistent within a classroom LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 • Let’s do an intervention: Hours of use of math tutoring software • Which level(s) of the model could this be at?
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 If students within a classroom varied in their tutor usage • Let’s do an intervention: Hours of use of math tutoring software • Which level(s) of the model could this be at?
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 • Can you find this out from R?
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS If classrooms within a school vary in tutor use, but consistent within a classroom LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 • Can you find this out from R? xtabs( ~ ClassroomID + TutorHours, data=tutor) •
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS If classrooms within a school vary in tutor use, but consistent within a classroom LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 • Try running a model to look at the effect of TutorHours on FinalMathScore • Hint: Don’t forget to account for the clustering! • Hint 2: You will need 2 random intercepts
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS If classrooms within a school vary in tutor use, but consistent within a classroom LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 model.Intercepts <- lmer(FinalMathScore ~ 1 + • TutorHours + (1|ClassroomID) + (1|School), data=tutor)
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS If classrooms within a school vary in tutor use, but consistent within a classroom LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 • Which do you think is better? • Teachers are allowed to choose how much they use the tutor in their classroom • We randomly assign a specific # of tutor hours
Multiple Random Effects LEVEL 3 School School Sampled SCHOOLS 1 2 LEVEL 2 Ms. Mr. Ms. Ms. Sampled Wagner’s Fulton’s Green’s Cornell’s Class Class Class Class CLASSROOMS If classrooms within a school vary in tutor use, but consistent within a classroom LEVEL 1 Student Student Student Student Sampled STUDENTS 1 2 3 4 • Random assignment allows stronger claims about causalit y • But, model is implemented the same either way • Just affects how we interpret the results
Introduction to Random Slopes • Right now, our assumption is that schools differ in their baseline math score (intercept) • What other ways might schools differ? • Some schools might use the tutor more effectively than others • Fidelity of implementation is a major concern in education research!
Introduction to Random Slopes • Overall relationship between a 100 ● ● ● ● classroom’s use of ● ● ● ● 90 ● ● the tutor & their ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Average final math score ● ● ● average math score ● ● ● ● ● ● ● 80 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 70 ● ● ● ● ● ● • Current assumption ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● is that the slope of ● ● ● ● ● ● ● ● ● ● ● 60 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● this line (= tutor ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 50 ● ● ● ● effect) is the same ● 10 15 20 25 30 35 in every school Hours that a classroom used the tutor
Assumption is that slope is Random Slopes the same for all schools • That is, so far, our model says that schools vary in baseline math score 70 70 70 • Random intercept • And that every 1 hour of 65 65 65 Average final math score tutor use ≈ 0.49 points Midland Crescent 60 60 60 gain in final math score • Slope is a fixed effect Highland 55 55 55 Different intercepts for each school 50 50 50 0 5 10 Hours that a classroom used the tutor
Slopes now also differ Random Slopes across sampled schools • That is, so far, our model says that schools vary in baseline math score 1200 1200 1200 1200 70 70 70 Subject 2 • Random intercept 1100 1100 1100 1100 • And that every 1 hour of 65 65 65 1000 1000 1000 1000 Subject 4 Average final math score tutor use ≈ 0.49 points Midland 900 900 900 900 RT Crescent gain in final math score 60 60 60 Subject 3 800 800 800 800 Schools that start off Subject 1 • Slope is a fixed effect worse show a bigger 700 700 700 700 Highland • Schools may also vary 55 55 55 benefit of tutor 600 600 600 600 Schools still vary in in how much tutor affects intercept 500 500 500 500 50 50 50 their classrooms’ scores 1 1 1 1 2 2 2 2 3 3 3 3 0 5 10 • A random slope of tutor Word frequency Hours that a classroom used the tutor usage by schools • Such differences may correlate with baseline diff.s
Random Slopes: Statistical Consequences • Important to capture this variability in the model • Observations from the same school will show more similar effects of tutor use • Different from what other schools would be like • If we don’t account Highland School Highland School for this similarity, (slope 0.82) (slope 0.82) non-independence Classroom 21, 27 hrs Classroom 22, 28 hrs that underestimates Highland School Highland School true variability (slope 0.82) (slope 0.82) Classroom 29, 25 hrs Classroom 30, 21 hrs Estimate t = Midland School Midland School Std. error (slope 0.17) (slope 0.17) Classroom 97, 27 hrs Classroom 98, 28 hrs • Inflates false positive rate
Week 5: Random Slopes l Finish Nested Random Effects l Recap l Level-2 Variables l Multiple Random Effects l Random Slopes l Introduction l Notation l Implementation l Testing Random Effects l Crossed Random Effects l Examples l Random Slopes, Revisited l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?
Notation • • Level-2 model of classroom j : = + + β 0 jk γ 000 γ 100 x 1 j ( k ) U 0j(k) Tutor hours Overall baseline Classroom Teacher effect for this intercept classroom (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) End-of-year math Baseline for students in Student exam score this classroom Error
Notation • Level-3 model of school k : = + δ 00k γ 000 V 00k School intercept School effect for this Overall baseline school (Error) • Level-2 model of classroom j : = + + β 0 jk δ 00k γ 100 x 1 j ( k ) U 0j(k) Tutor hours Baseline for classrooms Classroom Teacher effect for this in this school intercept classroom (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) End-of-year math Baseline for students in Student exam score this classroom Error
Notation Right now, tutor slope γ 100 is still just a fixed value. Maybe, like the intercept, we should allow to vary across schools. • Level-3 model of school k : = + δ 00k γ 000 V 00k School intercept School effect for this Overall baseline school (Error) • Level-2 model of classroom j : = + + β 0 jk δ 00k γ 100 x 1 j ( k ) U 0j(k) Tutor hours Baseline for classrooms Classroom Teacher effect for this in this school intercept classroom (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) End-of-year math Baseline for students in Student exam score this classroom Error
• Level-3 model of school k : = + δ 10k γ 100 V 10k Tutor slope By-school adjustment Overall tutor slope = for tutor slope (Error) + (fixed effect) δ 00k γ 000 V 00k School intercept School effect for this Overall baseline • Level-2 model of classroom j : school (Error) = + + β 0 jk δ 00k δ 10k x 1 j ( k ) U 0j(k) Tutor hours Baseline for classrooms Classroom Teacher effect for this in this school intercept classroom (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) End-of-year math Baseline for students in Student exam score this classroom Error
• Level-3 model of school k : = + δ 10k γ 100 V 10k Tutor slope By-school adjustment Overall tutor slope = for tutor slope (Error) + (fixed effect) δ 00k γ 000 V 00k School intercept School effect for this Overall baseline • Level-2 model of classroom j : school (Error) = + + β 0 jk δ 00k δ 10k x 1 j ( k ) U 0j(k) Tutor hours Baseline for classrooms Classroom Teacher effect for this in this school intercept classroom (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) Let’s start doing our algebraic End-of-year math Baseline for students in Student substitution exam score this classroom Error
• Level-3 model of school k : = + δ 10k γ 100 V 10k Tutor slope By-school adjustment Overall tutor slope = for tutor slope (Error) + (fixed effect) δ 00k γ 000 V 00k School intercept School effect for this Overall baseline • Level-2 model of classroom j : school (Error) = + + β 0 jk δ 00k δ 10k x 1 j ( k ) U 0j(k) Tutor hours Baseline for classrooms Classroom Teacher effect for this in this school intercept classroom (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) Let’s start doing our algebraic End-of-year math Baseline for students in Student substitution exam score this classroom Error
• Level-3 model of school k : = + δ 10k γ 100 V 10k Tutor slope By-school adjustment Overall tutor slope for tutor slope (Error) (fixed effect) • Level-2 model of classroom j : = + + β 0 jk δ 10k x 1 j ( k ) γ 000 U 0j(k) Tutor hours + Overall baseline Classroom Teacher effect for this V 00k intercept classroom (Error) By-school adjustment for intercept (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) Let’s start doing our algebraic End-of-year math Baseline for students in Student substitution exam score this classroom Error
• Level-3 model of school k : = + δ 10k γ 100 V 10k Tutor slope By-school adjustment Overall tutor slope for tutor slope (Error) (fixed effect) • Level-2 model of classroom j : = + + β 0 jk δ 10k x 1 j ( k ) γ 000 U 0j(k) Tutor hours + Overall baseline Classroom Teacher effect for this V 00k intercept classroom (Error) By-school adjustment for intercept (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) Let’s start doing our algebraic End-of-year math Baseline for students in Student substitution exam score this classroom Error
• Level-2 model of classroom j : = + β 0 jk γ 000 (γ 100 + V 10k ) x 1 j ( k ) Tutor hours effect Overall baseline Classroom (overall & by school) intercept + + V 00k U 0j(k) By-school adjustment Teacher effect for this for intercept (Error) classroom (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) Let’s start doing our algebraic End-of-year math Baseline for students in Student substitution exam score this classroom Error
• Level-2 model of classroom j : = + β 0 jk γ 000 (γ 100 + V 10k ) x 1 j ( k ) Tutor hours effect Overall baseline Classroom (overall & by school) intercept + + V 00k U 0j(k) By-school adjustment Teacher effect for this for intercept (Error) classroom (Error) • Level 1 model of student i : = + Y i(jk) β 0 jk E i ( jk) Let’s start doing our algebraic End-of-year math Baseline for students in Student substitution exam score this classroom Error
• Mixed-effects model of student i : + = γ 000 (γ 100 + V 10k ) x 1 j ( k ) Y i(jk) Tutor hours effect Overall baseline End-of-year math (overall & by school) exam score + + V 00k U 0j(k) By-school adjustment Teacher effect for this for intercept (Error) classroom (Error) + E i ( jk) Student Error
• Mixed-effects model of student i : Apply distributive property + = γ 000 (γ 100 + V 10k ) x 1 j ( k ) Y i(jk) Tutor hours effect Overall baseline End-of-year math (overall & by school) exam score + + V 00k U 0j(k) By-school adjustment Teacher effect for this for intercept (Error) classroom (Error) + E i ( jk) Student Error
• Mixed-effects model of student i : + γ 100 x 1 j ( k ) = γ 000 Y i(jk) FIXED Fixed effect of tutor hours Overall baseline End-of-year math (overall) exam score + + + U 0j(k) V 10k x 1j(k) V 00k RANDOM By-school adjustment By-school adjustment Teacher effect for this for tutor slope (Error) for intercept (Error) classroom (Error) + E i ( jk) Student Error
Covariance Matrix • Because we have more than one random variable, they can also covary [ The correlation Variance of subject ] parameter! intercept Covariance σ 2u00 j 0 cov( σ 2u00 j 0, u10 j 0 ) matrix for subject random cov( σ 2u00 j 0, u10 j 0 ) σ 2u10 j 0 effects in maximal model Variance of TutorHours slope across schools [ ] In near- σ 2u00 j 0 0 maximal model without the 0 σ 2u10 j 0 correlation parameter
Week 5: Random Slopes l Finish Nested Random Effects l Recap l Level-2 Variables l Multiple Random Effects l Random Slopes l Introduction l Notation l Implementation l Testing Random Effects l Crossed Random Effects l Examples l Random Slopes, Revisited l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?
Random Slopes: Implementation + γ 100 x 1 j ( k ) = γ 000 Y i(jk) FIXED Fixed effect of tutor hours Overall baseline End-of-year math (overall) exam score + + + U 0j(k) V 10k x 1j(k) V 00k RANDOM By-school adjustment By-school adjustment Teacher effect for this for tutor slope (Error) for intercept (Error) classroom (Error) • So, lmer model: model.Slope <- lmer(FinalMathScore ~ • Next, we need overall (fixed) effects of Intercept and TutorHours
Random Slopes: Implementation + γ 100 x 1 j ( k ) = γ 000 Y i(jk) FIXED Fixed effect of tutor hours Overall baseline End-of-year math (overall) exam score + + + U 0j(k) V 10k x 1j(k) V 00k RANDOM By-school adjustment By-school adjustment Teacher effect for this for tutor slope (Error) for intercept (Error) classroom (Error) • So, lmer model: model.Slope <- lmer(FinalMathScore ~ 1 + • TutorHours Next, we need overall (fixed) effects of Intercept and TutorHours
Random Slopes: Implementation + γ 100 x 1 j ( k ) = γ 000 Y i(jk) FIXED Fixed effect of tutor hours Overall baseline End-of-year math (overall) exam score + + + U 0j(k) V 10k x 1j(k) V 00k RANDOM By-school adjustment By-school adjustment Teacher effect for this for tutor slope (Error) for intercept (Error) classroom (Error) • So, lmer model: model.Slope <- lmer(FinalMathScore ~ 1 + • TutorHours How about a random difference in the intercept for each ClassroomID?
Random Slopes: Implementation + γ 100 x 1 j ( k ) = γ 000 Y i(jk) FIXED Fixed effect of tutor hours Overall baseline End-of-year math (overall) exam score + + + U 0j(k) V 10k x 1j(k) V 00k RANDOM By-school adjustment By-school adjustment Teacher effect for this for tutor slope (Error) for intercept (Error) classroom (Error) • So, lmer model: model.Slope <- lmer(FinalMathScore ~ 1 + • TutorHours + (1|ClassroomID) How about a random difference in the intercept for each ClassroomID?
Random Slopes: Implementation + γ 100 x 1 j ( k ) = γ 000 Y i(jk) FIXED Fixed effect of tutor hours Overall baseline End-of-year math (overall) exam score + + + U 0j(k) V 10k x 1j(k) V 00k RANDOM By-school adjustment By-school adjustment Teacher effect for this for tutor slope (Error) for intercept (Error) classroom (Error) • So, lmer model: model.Slope <- lmer(FinalMathScore ~ 1 + • TutorHours + (1|ClassroomID) Lastly, both intercept and TutorHours slope need to vary across schools
Random Slopes: Implementation + γ 100 x 1 j ( k ) = γ 000 Y i(jk) FIXED Fixed effect of tutor hours Overall baseline End-of-year math (overall) exam score + + + U 0j(k) V 10k x 1j(k) V 00k RANDOM By-school adjustment By-school adjustment Teacher effect for this for tutor slope (Error) for intercept (Error) classroom (Error) • So, lmer model: model.Slope <- lmer(FinalMathScore ~ 1 + • TutorHours + (1|ClassroomID) + (1+TutorHours|School) Lastly, both intercept and TutorHours slope need to vary across schools
Random Slopes: Implementation + γ 100 x 1 j ( k ) = γ 000 Y i(jk) FIXED Fixed effect of tutor hours Overall baseline End-of-year math (overall) exam score + + + U 0j(k) V 10k x 1j(k) V 00k RANDOM By-school adjustment By-school adjustment Teacher effect for this for tutor slope (Error) for intercept (Error) classroom (Error) • So, lmer model: model.Slope <- lmer(FinalMathScore ~ 1 + • TutorHours + (1|ClassroomID) + (1+TutorHours|School), data=tutor)
Random Slopes: Implementation • Here’s our final model again: model.Slope <- lmer(FinalMathScore ~ • 1 + TutorHours + (1|ClassroomID) + (1 + TutorHours|School), data=tutor) Like a miniature model formula for things we think will vary by schools Schools differ in their Schools differ in the intercept (baseline math effectiveness of the tutor score) on their math scores
Random Slopes: Output NEW: How much schools vary from the mean Correlation: Schools TutorHours slope with a lower starting score show a larger benefit of the tutor Still have a fixed effect of TutorHours— slope estimated across all subjects & classrooms The fixed effect is reliable. In this case, we can conclude that final math scores are generally higher for classrooms that used the tutor more, even if there are some differences across schools in the size of this effect.
Random Slopes: Model Comparison As compared to the intercept-only model, our estimate of baseline school variance has now decreased—many of these unexplained “baseline” differences have now been explained as differing effectiveness of the tutor OLD MODEL (INTERCEPT ONLY) NEW MODEL (WITH SLOPE) • Now that we’ve properly accounted for clustering, our estimate of the standard error of the tutor effect is higher … and t value lower • Excluding the slope would have increased our Type I error rate
Random Effects: Implementation • By the way, how come we never discussed a (1+TutorHours|Classroom) slope? • Within a school , we Class 28 can calculate a 85 Class 25 ● ● regression line Class 24 ● Class 27 80 Average final math score ● Class 21 relating tutor use to ● Class 23 ● Class 29 ● Class 22 average math score ● 75 Class 26 ● • Each class = 1 point 70 Class 30 ● 65 20 22 24 26 28 30 32 Hours that a classroom used the tutor
Random Effects: Implementation • By the way, how come we never discussed a (1+TutorHours|Classroom) slope? • But each class has only a single value of 85 TutorHours 80 • Classroom 22 used it Average final math score for 28 hours Class 22 ● 75 • No way to draw a line relating different values 70 of TutorHours to score 65 • Need 2 points for a line! 20 22 24 26 28 30 32 Hours that a classroom used the tutor
Week 5: Random Slopes l Finish Nested Random Effects l Recap l Level-2 Variables l Multiple Random Effects l Random Slopes l Introduction l Notation l Implementation l Testing Random Effects l Crossed Random Effects l Examples l Random Slopes, Revisited l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?
Testing Random Effects • Does this random slope contribute significantly? • i.e., is there significant variation in the effectiveness of the tutor across schools? • We can compare the fit of models with & without the random slope • Using the likelihood-ratio test • Same as when we fixed effects last week
Testing Random Effects • Does this random slope contribute significantly? • i.e., is there significant variation in the effectiveness of the tutor across schools? • We can compare the fit of models with & without the random slope • Using the likelihood-ratio test • Same as when we tested fixed effects last week anova(model.Intercepts, model.Slope) • • Could test random intercepts the same way Model with slope fits significantly better!
Testing Random Effects • Remember that this tests whether there is significant variation in the slope across schools • i.e., more than expected under H 0 of no variation • More complex model will always fit numerically the same or better • But if three schools had slopes of 0.39, 0.38, and 0.40, this would probably not be significant • Consistent with what’s expected from sampling error
Testing Random Effects • Caveats: • Overfitting / shrinkage • Problem we talked about last week about using the data to decide the model. “Using up” degrees of freedom • A new sample might yield a different random effects structure • In cases where we have a clear sampling design, would want model to reflect that • e.g., weird not to include random intercepts of classroom & school here
Week 5: Random Slopes l Finish Nested Random Effects l Recap l Level-2 Variables l Multiple Random Effects l Random Slopes l Introduction l Notation l Implementation l Testing Random Effects l Crossed Random Effects l Examples l Random Slopes, Revisited l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?
An Experimental Dataset • naming.csv • RT to name/read a word aloud for students learning English • 60 Subjects each presented with 49 words (Items) we randomly picked out of a dictionary • Each row is a single trial (one subject responding to one word) • We’re interested in YearsOfStudy (of English) and WordFreq (word frequency) • Also interested in their interaction • Which of these should we consider Fixed Effects? Which are Random Effects?
An Experimental Dataset • naming.csv • RT to name/read a word aloud for students learning English • 60 Subjects each presented with 49 words (Items) we randomly picked out of a dictionary • Each row is a single trial (one subject responding to one word) • We’re interested in YearsOfStudy (of English) and WordFreq (word frequency) • Also interested in their interaction • Which of these should we consider Fixed Effects? Which are Random Effects? • Fixed: YearsOfStudy, WordFreq, & interaction • Random: Subject, Item
Crossed Random Effects Subject Subject Subjects (Level-2) 1 2 RT 1 RT 2 RT 3 RT 4 Trials (Level-1) • Now, we have >1 trial per Subject • Level-1 observations (RTs) are nested within subjects • Clear we need to take account of this • Some subjects will have faster RTs in general than others
Crossed Random Effects • Try an R command to look at the categories of the Item variable. Do some seem easier than others?
Crossed Random Effects • Try an R command to look at the categories of the Item variable. Do some seem easier than others? summary(naming$Item) • Probably relatively Probably relatively easy hard!
Crossed Random Effects Subject Subject Subjects (Level-2) 1 2 RT 1 RT 2 RT 3 RT 4 Trials (Level-1) “Boy” “Carbu Words retor” • Each word is also used in more than 1 trial • Want to take account of that, too • Observations from the same item will be more similar to one another, too (“boy” easier than “carburetor”)
Crossed Random Effects Subject Subject Subjects (Level-2) 1 2 RT 1 RT 2 RT 3 RT 4 Trials (Level-1) “Boy” “Carbu Words (Level-2) retor” • In fact, we can think of each trial as the pairing of a subject and a word
Crossed Random Effects Subject Subject Subjects (Level-2) 1 2 RT 1 RT 2 RT 3 RT 4 Trials (Level-1) “Boy” “Carbu Words (Level-2) retor” • Random effects not hierarchically nested • Before: Each classroom appears in only 1 school • Here: Each item presented to each subject • If we draw all the lines in, we see they cross
Crossed Random Effects Subject Subject Subjects (Level-2) 1 2 RT 1 RT 2 RT 3 RT 4 Trials (Level-1) “Boy” “Carbu Words (Level-2) retor” • Crossed random effects structure • a/k/a cross-classified when we’re dealing with existing classifications
Crossed Random Effects Subject Subject Subjects (Level-2) 1 2 RT 1 RT 2 RT 3 RT 4 Trials (Level-1) “Boy” “Carbu Words (Level-2) retor” • Conceptually different sampling, but same syntax! • Try fitting a model with: • Fixed effects of YearsOfStudy , WordFreq , & interaction • Random intercepts for Subject and Item
Crossed Random Effects model1 <- lmer(RT ~ 1+ YearsOfStudy * WordFreq + • (1|Subject) + (1|Item), data=naming) Greater Subject variance than Item variance Typical in experiments because we often design items to be similar 2 significant main effects, but no interaction
Crossed Random Effects Subject Subject Subjects (Level-2) 1 2 RT 1 RT 2 RT 3 RT 4 Trials (Level-1) “Boy” “Carbu Words (Level-2) retor” • Huge improvement over ANOVA analyses in psycholinguistics / experimental psychology! • We used to have to conduct separate analyses to assess generalizability over subjects & items • Or, item effects were simply ignored!
Week 5: Random Slopes l Finish Nested Random Effects l Recap l Level-2 Variables l Multiple Random Effects l Random Slopes l Introduction l Notation l Implementation l Testing Random Effects l Crossed Random Effects l Examples l Random Slopes, Revisited l Between-Subjects & Within-Subjects Designs l Between-Items & Within-Items Designs l When are Random Slopes Necessary?
Between vs Within Subjects • What kind of random slopes might be relevant here? • To answer this question, we need to understand how “between-subjects variables” differ from “within-subjects variables”
Between vs Within Subjects • In an experimental design, some variables are: Ø Between-Subjects variables : Each subject is in 1 and only 1 group … or has 1 and only 1 value • Randomly assigned to drug 1 vs. drug 2 vs. placebo • Demographic variables; e.g., SES • Cognitive/linguistic differences (e.g., working mem. score) • “Between subjects” because differences in this variable are only seen between one subject and another SUBJECT 10’s SUBJECT 11’s DATA DATA Native speaker Non-native speaker Trial 1: Correct Trial 1: Incorrect Trial 2: Correct Trial 2: Correct Trial 3: Incorrect Trial 3: Incorrect
Between vs Within Subjects • In an experimental design, some variables are: Ø Between-Subjects variables : Each subject is in 1 and only 1 group … or has 1 and only 1 value • Randomly assigned to drug 1 vs. drug 2 vs. placebo • Demographic variables; e.g., SES • Cognitive/linguistic differences (e.g., working mem. score) • “Between subjects” because differences in this variable are only seen between one subject and another Ø Within-Subjects variables: Same subject sees more than 1 condition or has >1 value • Same subject sees both congruent (green) and incongruent (blue) Stroop trials • Values that vary w/in a study, e.g., # of previous trials • Variables where you’d use a repeated measures ANOVA • “Within-subjects” because you can see differences in this variable even within a single subject
Between vs Within Subjects • In an experimental design, some variables are: Ø Between-Subjects variables : Each subject is in 1 SUBJECT 12’S DATA and only 1 group … or has 1 and only 1 value • Randomly assigned to @ Trial 1: Congruent Stroop, 655 ms Trial 2: Incongruent Stroop, 512 ms • Demographic variables; e.g., SES Trial 3: Incongruent Stroop, 711 ms • Cognitive/linguistic differences (e.g., working mem. score) Trial 4: Congruent Stroop: 642 ms • “Between subjects” because differences in this variable are only seen between one subject and another Ø Within-Subjects variables: Same subject sees more than 1 condition or has >1 value • Same subject sees both congruent (green) and incongruent (blue) Stroop trials • Values that vary w/in a study, e.g., # of previous trials • Variables where you’d use a repeated measures ANOVA • “Within-subjects” because you can see differences in this variable even within a single subject
Between vs Within: Advanced • The same variable could end up as between- or within-subjects, depending on experimental design • I’m interested in maintenance rehearsal (repetition) vs. elaborative rehearsal (relating to other concepts) • Half of my participants study second-language vocab using maintenance rehearsal, and half study words using elaborative rehearsal à Between Subjects • Each participant studies some words with maintenance rehearsal and some with elaborative rehearsal à Within Subjects
Between vs Within: Practice • I assign 40 students to study for an upcoming biology exam by practicing retrieving the facts ( retrieval practice ) and 40 students to re-read the textbook ( restudy ) . Practice Type is… • In a visual perception task, 20 subjects try to determine as quickly as possible whether a circle is present on the screen (among other objects). Each subject sees ½ of the trials where the square is present and ½ where it’s not. Trial Type is… • Kevin compares 100 first-generation college students to 100 college students who are not first- generation students in their feeling of belonging at their university. College history is…
Between vs Within: Practice • I assign 40 students to study for an upcoming biology exam by practicing retrieving the facts ( retrieval practice ) and 40 students to re-read the textbook ( restudy ) . Practice Type is… • Between subjects • In a visual perception task, 20 subjects try to determine as quickly as possible whether a circle is present on the screen (among other objects). Each subject sees ½ of the trials where the square is present and ½ where it’s not. Trial Type is… • Within subjects • Kevin compares 100 first-generation college students to 100 college students who are not first- generation students in their feeling of belonging at their university. College history is… • Between subjects
Between vs Within: More Practice • I recruit 40 Psych Subject Pool students for my experiment. Each participant reads 20 syntactically ambiguous sentences and 20 unambiguous sentences. Sentence type is… • I recruit 40 Psych Subject Pool students for my experiment. 20 participants read syntactically ambiguous sentences, and the other 20 read only unambiguous sentences. Sentence type is…
Between vs Within: More Practice • I recruit 40 Psych Subject Pool students for my experiment. Each participant reads 20 syntactically ambiguous sentences and 20 unambiguous sentences. Sentence type is… • Within subjects • I recruit 40 Psych Subject Pool students for my experiment. 20 participants read syntactically ambiguous sentences, and the other 20 read only unambiguous sentences. Sentence type is… • Between subjects
Between vs Within: More Practice • Each of 100 participants in my study makes a decision in each of 2 different moral dilemmas: one involving direct harm, and one involving indirect harm. I also measure participants’ working memory and split them into “low WM” and “high WM” groups. • Moral dilemma type is… • Working memory is…
Between vs Within: More Practice • Each of 100 participants in my study makes a decision in each of 2 different moral dilemmas: one involving direct harm, and one involving indirect harm. I also measure participants’ working memory and split them into “low WM” and “high WM” groups. • Moral dilemma type is… • Within subjects • Working memory is… • Between subjects
Between vs Within: More Practice • How about in our naming.csv dataset? • Word frequency is… • Each subject sees high-, medium- and low-frequency words • Within subjects • Years of study is… • This experiment takes place at a single point in time, so the number of years a subject has been studying English is fixed and never varies • Between subjects
When are Random Slopes Appropriate? • If the random effect is subject… • Random slopes for subjects appropriate for: • Within-subjects variables • We can draw a regression line for each subject • Calculate the effect “within” each subject Subject 1 High Frequency Subject 1 Low Frequency Subject 2 High Frequency Subject 2 Low Frequency • Random slopes for subjects inappropriate for: • Between-subjects variables • Can’t draw a regression line within each subject Subject 1: 6 years of study Subject 2: 2 years of study
Random Slopes: Implementation • Remember, original model was: model1 <- lmer(RT ~ 1 + WordFreq * YearsOfStudy • + (1|Subject) + (1|Item), data=naming) • What variable varies within subjects? Try adding a random slope for it
Random Slopes: Implementation • Remember, original model was: model1 <- lmer(RT ~ 1 + WordFreq * YearsOfStudy • + (1|Subject) + (1|Item), data=naming) • With slope: model2 <- lmer(RT ~ 1 + WordFreq * YearsOfStudy • + (1+WordFreq|Subject) + (1|Item), data=naming) Again, miniature model formula for things we think will vary by subjects Subjects differ in their Subjects differ in the intercept (baseline RT) effect of word frequency on their RTs
Slope is the same for Random Slopes everyone • Original model says that subjects vary in baseline RT 1200 1200 1200 1200 Subject 2 • Random intercept 1100 1100 1100 1100 • And that a 1-unit change 1000 1000 1000 1000 Subject 4 in word frequency ≈ 120 900 900 900 900 RT ms decrease in RT Subject 3 800 800 800 800 Subject 1 • Slope is a fixed effect 700 700 700 700 600 600 600 600 Different intercepts for each subject 500 500 500 500 1 1 1 1 2 2 2 2 3 3 3 3 Word frequency
Recommend
More recommend
