Und Under erstanding standing Applicat plication ions s in t - PowerPoint PPT Presentation

Hi Hier erarc archical hical Line inear ar Mo Mode deling: ling: Und Under erstanding standing Applicat plication ions s in t in the e MS MSP Proje jects cts NSF # DRL1238120

The work of TEAMS is supported with funding provided by the National Science Foundation, Award Number DRL 1238120. Any opinions, suggestions, and conclusions or recommendations expressed in this presentation are those of the presenter and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content. 2

Strengthening the quality of the MSP project evaluation and building the capacity of the evaluators by strengthening their skills related to evaluation design, methodology, analysis, and reporting. 3

 Website at http tp://t //tea eams.msp ms.mspnet.org et.org  Online Help-Desk for submitting requests  Assistance with instruments  Consultation and targeted TA  Webinar series on specific evaluation topics  White papers/focused topic papers 4

Hie ierarchical rarchical Lin inear ar Model deling: ing: Under derst standing anding Applicat licatio ions ns in in the MSP SP Proj rojects cts Presen esenter ers: s: Karen Ka en Drill ill, RMC Research Corporation Emma mma Espel pel, RMC Research Corporation Moderat rator: or: John Sutt tton on, RMC Research Corporation, TEAMS Project PI 5

Goals: Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use of HLM within MSP projects Provide concrete examples of the use of HLM within MSP projects NSF # DRL1238120 6

Webinar Sections What is HLM? When to use HLM Example HLM Use Pro Tips: What (not) to do Tools & Resources 7

Webinar Sections What is HLM? When to use HLM Example HLM Use Pro Tips: What (not) to do Tools & Resources 8

What is HLM? How w fa fami miliar liar are you u with h HLM? 9

What is HLM? A complex form of ordinary least squares regressi gression on Can be used to analyze variance in outcome variables when predictor variables are at different hierar erarchical chical levels ls 10

Revie iew of Linear ear Regres ression sion Linear regression attempts to model the relationship ement ievemen between two variables by fit ittin ing g a a lin inea ear equa equation ion to h achie obser erved ed da data. Math Math h interest est 11

Revie iew of Linear ear Regres ression sion Y ’ = 𝐶 0 + 𝐶 𝑍𝑌 𝑌 + 𝜁 100% Y ’ = 0.002 + 0.180 𝑌 + .210 am t exam Y ’ = The predicted value ent h conten 𝐶 0 = Y -intercept — the value of Y ’ when X = 0 ect on math 𝐶 𝑍𝑌 = Slope — the regression coefficient for predicting Y 50% orrec X = Independent variable or predictor ent cor Percen 𝜁 = Error 0% 0 1 2 3 4 5 Student interest in math 12

Revie iew of Linear ear Regres ression sion Y ’ = 𝐶 0 + 𝐶 𝑍𝑌 𝑌 + 𝜁 100% Y ’ = 0.002 + 0.180 𝑌 + .210 am t exam Y ’ = The predicted value ent h conten 𝐶 0 = Y -intercept — the value of Y ’ when X = 0 ect on math 𝐶 𝑍𝑌 = Slope — the regression coefficient for predicting Y 50% orrec X = Independent variable or predictor ent cor Percen 𝜁 = Error 0% 0 1 2 3 4 5 Student interest in math 13

Revie iew of Linear ear Regres ression sion Y ’ = 𝐶 0 + 𝐶 𝑍𝑌 𝑌 + 𝜁 100% Y ’ = 0.002 + 0.180 𝑌 + .210 am t exam Y ’ = The predicted value ent h conten 𝐶 0 = Y -intercept — the value of Y ’ when X = 0 ect on math 𝐶 𝑍𝑌 = Slope — the regression coefficient for predicting Y 50% orrec X = Independent variable or predictor ent cor Percen 𝜁 = Error 0% 0 1 2 3 4 5 Student interest in math 14

Revie iew of Linear ear Regres ression sion Y ’ = 𝐶 0 + 𝐶 𝑍𝑌 𝑌 + 𝜁 100% Y ’ = 0.002 + 0.180 𝑌 + .210 am t exam Y ’ = The predicted value ent h conten 𝐶 0 = Y -intercept — the value of Y ’ when X = 0 ect on math 𝐶 𝑍𝑌 = Slope — the regression coefficient for predicting Y 50% orrec X = Independent variable or predictor ent cor Percen 𝜁 = Error 0% 0 1 2 3 4 5 Student interest in math 15

Revie iew of Linear ear Regres ression sion Y ’ = 𝐶 0 + 𝐶 𝑍𝑌 𝑌 + 𝜁 100% Y ’ = 0.002 + 0.180 𝑌 + .210 am t exam Y ’ = The predicted value ent h conten 𝐶 0 = Y -intercept — the value of Y ’ when X = 0 ect on math 𝐶 𝑍𝑌 = Slope — the regression coefficient for predicting Y 50% orrec X = Independent variable or predictor ent cor Percen 𝜁 = Error 0% 0 1 2 3 4 5 Student interest in math 16

HLM Simi imilarities larities to Linear ear Regress gression ion Based Ba sed on on linea near r reg egression ession 17

HLM Simi imilarities larities to Linear ear Regress gression ion Mo Models dels th the e rel elationship ationship bet between en th the e obs bser erved ed to th the exp xpect ected ed 18

HLM Simi imilarities larities to Linear ear Regress gression ion Can an be be cr cross ss-sect sectional ional or lo longitudinal gitudinal 19

Differen erences es from m Linear ear Regression gression Level-3 (school) Level-2 (teacher ) Level-1 (students ) Green een = Level 1 Orange nge = Level 2 20

Differen erences es from m Linear ear Regression gression Level-3 (school) Level-2 (teacher ) Level-1 (students ) Cluster 1 Cluster 2 Cluster 4 Cluster 5 Cluster 3 Intracl aclass ass Correlat ation on (ICC CC) 21

Differen erences es from m Linear ear Regression gression HLM: Multi tiple e Levels Ecological ogical Fallacy acy: One Level Green een = Level 1 Orange nge = Level 2 22

Questi tion ons? s? 23

MSP Scenario and HLM Equations 24

MSP Scenario and HLM Equations You are the evaluator of an MSP that is implementing an innovative math curriculum for 6 th graders. You are interested in whether implementing this curriculum influences students’ math achievement scores. Your sample also includes a matched comparison group of teachers not implementing the curriculum. To what extent does teacher implementation of the math curriculum influence students’ math achievement scores? 25

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores? Var ariab ables les Y = Students’ achievement scores (level-1 outcome) X = Female (level-1 predictor) W = Treatment (the math curriculum) (level-2 predictor) 26

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores? (level-1) 𝑍 𝑗𝑘 = 𝛾 0𝑘 + 𝛾 1𝑘 ( 𝐺𝑓𝑛𝑏𝑚𝑓 𝑗𝑘 )* + 𝑠 𝑗𝑘 𝑍 𝑗𝑘 = dependent variable measured for 𝑗 th level-1 (student) unit nested within the 𝑘 th level-2 (teacher) unit 𝑍 𝑗𝑘 = students’ math achievement score *dummy coded 27

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores? (level-1) 𝑍 𝑗𝑘 = 𝛾 0𝑘 + 𝛾 1𝑘 ( 𝐺𝑓𝑛𝑏𝑚𝑓 𝑗𝑘 ) + 𝑠 𝑗𝑘 𝛾 0𝑘 = intercept for the 𝑘 th level-2 (teacher) unit 𝛾 0𝑘 = best estimate for predicting math achievement for males 28

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores? (level-1) 𝑍 𝑗𝑘 = 𝛾 0𝑘 + 𝛾 1𝑘 ( 𝐺𝑓𝑛𝑏𝑚𝑓 𝑗𝑘 ) + 𝑠 𝑗𝑘 𝛾 1𝑘 = regression coefficient associated with 𝑌 𝑗𝑘 for the 𝑘 th level-2 (teacher) unit 𝛾 1𝑘 = level-1 slope 𝛾 1𝑘 = the effect of being female on math achievement 29

To what extent does teacher implementation of the math curriculum influence students’ math content achievement scores? (level-1) 𝑍 𝑗𝑘 = 𝛾 0𝑘 + 𝛾 1𝑘 ( 𝐺𝑓𝑛𝑏𝑚𝑓 𝑗𝑘 ) + 𝑠 𝑗𝑘 (𝐺𝑓𝑛𝑏𝑚𝑓) 𝑗𝑘 = value on the level-1 (student) predictor (𝐺𝑓𝑛𝑏𝑚𝑓 𝑗𝑘 ) = value for female (0 = not female, 1 = female)* *dummy coded 30

To what extent does teacher implementation of the math curriculum influence students’ math content knowledge? (level-1) 𝑍 𝑗𝑘 = 𝛾 0𝑘 + 𝛾 1𝑘 ( 𝐺𝑓𝑛𝑏𝑚𝑓 𝑗𝑘 ) + 𝑠 𝑗𝑘 𝑠 𝑗𝑘 = random error associated with the 𝑗 th level-1 unit (student) nested within the 𝑘 th level-2 (teacher) unit 𝑠 𝑗𝑘 = deviation for each student from the fitted model 31

To what extent does teacher implementation of the math curriculum influence students’ math content knowledge? (level-1) 𝑍 𝑗𝑘 = 𝛾 0𝑘 + 𝛾 1𝑘 ( 𝐺𝑓𝑛𝑏𝑚𝑓 𝑗𝑘 ) + 𝑠 𝑗𝑘 (level -2) 𝛾 0𝑘 = 𝛿 00 + 𝛿 01 (𝑈𝑦) 1𝑘 + 𝑣 0𝑘 𝛾 0𝑘 = intercept for the 𝑘 th level-2 unit 32

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores? (level-1) 𝑍 𝑗𝑘 = 𝛾 0𝑘 + 𝛾 1𝑘 ( 𝐺𝑓𝑛𝑏𝑚𝑓 𝑗𝑘 ) + 𝑠 𝑗𝑘 (level -2) 𝛾 0𝑘 = 𝛿 00 + 𝛿 01 (𝑈𝑦) 1𝑘 + 𝑣 0𝑘 Υ 00 = level-2 intercept Υ 00 = mean math achievement for comparison schools 33