Und Under erstanding standing Applicat plication ions s in t - - PowerPoint PPT Presentation

β–Ά
und under erstanding standing applicat plication ions s
SMART_READER_LITE
LIVE PREVIEW

Und Under erstanding standing Applicat plication ions s in t - - PowerPoint PPT Presentation

Aug 08, 2023 347 likes β€’1.28k views

Hi Hier erarc archical hical Line inear ar Mo Mode deling: ling: Und Under erstanding standing Applicat plication ions s in t in the e MS MSP Proje jects cts NSF # DRL1238120 The work of TEAMS is supported with funding

slide-1
SLIDE 1

Hi Hier erarc archical hical Line inear ar Mo Mode deling: ling: Und Under erstanding standing Applicat plication ions s in t in the e MS MSP Proje jects cts

NSF # DRL1238120

slide-2
SLIDE 2

The work of TEAMS is supported with funding provided by the National Science Foundation, Award Number DRL

  • 1238120. Any opinions, suggestions, and conclusions or

recommendations expressed in this presentation are those

  • f the presenter and do not necessarily reflect the views of

the National Science Foundation; NSF has not approved or endorsed its content.

2

slide-3
SLIDE 3

Strengthening the quality of the MSP project evaluation and building the capacity of the evaluators by strengthening their skills related to evaluation design, methodology, analysis, and reporting.

3

slide-4
SLIDE 4
  • Website at http

tp://t //tea eams.msp ms.mspnet.org et.org

  • Online Help-Desk for submitting requests
  • Assistance with instruments
  • Consultation and targeted TA
  • Webinar series on specific evaluation topics
  • White papers/focused topic papers

4

slide-5
SLIDE 5

Hie ierarchical rarchical Lin inear ar Model deling: ing: Under derst standing anding Applicat licatio ions ns in in the MSP SP Proj rojects cts

Presen esenter ers: s: Ka Karen en Drill ill, RMC Research Corporation Emma mma Espel pel, RMC Research Corporation Moderat rator:

  • r:

John Sutt tton

  • n, RMC Research Corporation,

TEAMS Project PI

5

slide-6
SLIDE 6

6

Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use of HLM within MSP projects Provide concrete examples of the use of HLM within MSP projects

Goals:

NSF # DRL1238120

slide-7
SLIDE 7

7

What is HLM? When to use HLM Example HLM Use Pro Tips: What (not) to do

Webinar Sections

Tools & Resources

slide-8
SLIDE 8

8

What is HLM? When to use HLM Example HLM Use Pro Tips: What (not) to do

Webinar Sections

Tools & Resources

slide-9
SLIDE 9

9

What is HLM?

How w fa fami miliar liar are you u with h HLM?

slide-10
SLIDE 10

A complex form of

  • rdinary least squares

regressi gression

  • n

Can be used to analyze variance in outcome variables when predictor variables are at different hierar erarchical chical levels ls

10

What is HLM?

slide-11
SLIDE 11

Linear regression attempts to model the relationship between two variables by fit ittin ing g a a lin inea ear equa equation ion to

  • bser

erved ed da data.

11

Revie iew of Linear ear Regres ression sion

Math h interest est Math h achie ievemen ement

slide-12
SLIDE 12

Y’= 𝐢0 + πΆπ‘π‘Œπ‘Œ + 𝜁

Y’ = The predicted value 𝐢0 = Y-interceptβ€”the value of Y’ when X = 0 πΆπ‘π‘Œ = Slopeβ€”the regression coefficient for predicting Y X = Independent variable or predictor 𝜁 = Error

12

Revie iew of Linear ear Regres ression sion

0% 50% 100% 1 2 3 4 5

Y’= 0.002 + 0.180 π‘Œ + .210 Student interest in math Percen ent cor

  • rrec

ect on math h conten ent t exam am

slide-13
SLIDE 13

Y’= 𝐢0 + πΆπ‘π‘Œπ‘Œ + 𝜁

Y’ = The predicted value 𝐢0 = Y-interceptβ€”the value of Y’ when X = 0 πΆπ‘π‘Œ = Slopeβ€”the regression coefficient for predicting Y X = Independent variable or predictor 𝜁 = Error

13

Revie iew of Linear ear Regres ression sion

0% 50% 100% 1 2 3 4 5

Y’= 0.002 + 0.180 π‘Œ + .210 Student interest in math Percen ent cor

  • rrec

ect on math h conten ent t exam am

slide-14
SLIDE 14

Y’= 𝐢0 + πΆπ‘π‘Œπ‘Œ + 𝜁

Y’ = The predicted value 𝐢0 = Y-interceptβ€”the value of Y’ when X = 0 πΆπ‘π‘Œ = Slopeβ€”the regression coefficient for predicting Y X = Independent variable or predictor 𝜁 = Error

14

Revie iew of Linear ear Regres ression sion

0% 50% 100% 1 2 3 4 5

Y’= 0.002 + 0.180 π‘Œ + .210 Student interest in math Percen ent cor

  • rrec

ect on math h conten ent t exam am

slide-15
SLIDE 15

Y’= 𝐢0 + πΆπ‘π‘Œπ‘Œ + 𝜁

Y’ = The predicted value 𝐢0 = Y-interceptβ€”the value of Y’ when X = 0 πΆπ‘π‘Œ = Slopeβ€”the regression coefficient for predicting Y X = Independent variable or predictor 𝜁 = Error

15

Revie iew of Linear ear Regres ression sion

0% 50% 100% 1 2 3 4 5

Y’= 0.002 + 0.180 π‘Œ + .210 Student interest in math Percen ent cor

  • rrec

ect on math h conten ent t exam am

slide-16
SLIDE 16

Y’= 𝐢0 + πΆπ‘π‘Œπ‘Œ + 𝜁

Y’ = The predicted value 𝐢0 = Y-interceptβ€”the value of Y’ when X = 0 πΆπ‘π‘Œ = Slopeβ€”the regression coefficient for predicting Y X = Independent variable or predictor 𝜁 = Error

16

Revie iew of Linear ear Regres ression sion

0% 50% 100% 1 2 3 4 5

Y’= 0.002 + 0.180 π‘Œ + .210 Student interest in math Percen ent cor

  • rrec

ect on math h conten ent t exam am

slide-17
SLIDE 17

Ba Based sed on

  • n linea

near r reg egression ession

17

HLM Simi imilarities larities to Linear ear Regress gression ion

slide-18
SLIDE 18

Mo Models dels th the e rel elationship ationship bet between en th the e

  • bs

bser erved ed to th the exp xpect ected ed

18

HLM Simi imilarities larities to Linear ear Regress gression ion

slide-19
SLIDE 19

Can an be be cr cross ss-sect sectional ional or lo longitudinal gitudinal

19

HLM Simi imilarities larities to Linear ear Regress gression ion

slide-20
SLIDE 20

20

Differen erences es from m Linear ear Regression gression

Level-3 (school) Level-2 (teacher) Level-1 (students)

Green een = Level 1 Orange nge = Level 2

slide-21
SLIDE 21

21

Differen erences es from m Linear ear Regression gression

Level-3 (school) Level-2 (teacher) Level-1 (students)

Intracl aclass ass Correlat ation

  • n (ICC

CC)

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5

slide-22
SLIDE 22

22

Differen erences es from m Linear ear Regression gression

HLM: Multi tiple e Levels Ecological

  • gical Fallacy

acy: One Level

Green een = Level 1 Orange nge = Level 2

slide-23
SLIDE 23

23

Questi tion

  • ns?

s?

slide-24
SLIDE 24

24

MSP Scenario and HLM Equations

slide-25
SLIDE 25

25

MSP Scenario and HLM Equations

You are the evaluator of an MSP that is implementing an innovative math curriculum for 6th graders. You are interested in whether implementing this curriculum influences students’ math achievement scores. Your sample also includes a matched comparison group of teachers not implementing the curriculum. To what extent does teacher implementation of the math curriculum influence students’ math achievement scores?

slide-26
SLIDE 26

Var ariab ables les

Y = Students’ achievement scores (level-1 outcome) X = Female (level-1 predictor) W = Treatment (the math curriculum) (level-2 predictor)

26

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores?

slide-27
SLIDE 27

(level-1)𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜)* + 𝑠 π‘—π‘˜

𝑍

π‘—π‘˜= dependent variable measured for 𝑗th level-1 (student) unit

nested within the π‘˜th level-2 (teacher) unit 𝑍

π‘—π‘˜ = students’ math achievement score

27

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores?

*dummy coded

slide-28
SLIDE 28

(level-1)𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + π‘ π‘—π‘˜

𝛾0π‘˜ = intercept for the π‘˜th level-2 (teacher) unit 𝛾0π‘˜ = best estimate for predicting math achievement for males

28

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores?

slide-29
SLIDE 29

(level-1)𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + π‘ π‘—π‘˜

𝛾1π‘˜ = regression coefficient associated with π‘Œπ‘—π‘˜ for the π‘˜th level-2 (teacher) unit 𝛾1π‘˜ = level-1 slope 𝛾1π‘˜ = the effect of being female on math achievement

29

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores?

slide-30
SLIDE 30

(level-1)𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + π‘ π‘—π‘˜

(πΊπ‘“π‘›π‘π‘šπ‘“)π‘—π‘˜ = value on the level-1 (student) predictor (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) = value for female (0 = not female, 1 = female)*

*dummy coded

30

To what extent does teacher implementation of the math curriculum influence students’ math content achievement scores?

slide-31
SLIDE 31

(level-1)𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + π‘ π‘—π‘˜

𝑠

π‘—π‘˜ = random error associated with the 𝑗th level-1 unit (student)

nested within the π‘˜th level-2 (teacher) unit 𝑠

π‘—π‘˜ = deviation for each student from the fitted model

31

To what extent does teacher implementation of the math curriculum influence students’ math content knowledge?

slide-32
SLIDE 32

(level-1) 𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + 𝑠 π‘—π‘˜

(level -2) 𝛾0π‘˜= 𝛿00 + 𝛿01 (π‘ˆπ‘¦)1π‘˜ + 𝑣0π‘˜

𝛾0π‘˜ = intercept for the π‘˜th level-2 unit

32

To what extent does teacher implementation of the math curriculum influence students’ math content knowledge?

slide-33
SLIDE 33

(level-1) 𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + 𝑠 π‘—π‘˜

(level -2) 𝛾0π‘˜= 𝛿00 + 𝛿01 (π‘ˆπ‘¦)1π‘˜ + 𝑣0π‘˜

Ξ₯00 = level-2 intercept Ξ₯00 = mean math achievement for comparison schools

33

To what extent does teacher implementation of the math curriculum influence students’ math achievement scores?

slide-34
SLIDE 34

(level-1) 𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + 𝑠 π‘—π‘˜

(level -2) 𝛾0π‘˜= 𝛿00 + 𝛿01 (π‘ˆπ‘¦)1π‘˜ + 𝑣0π‘˜

Ξ₯01 = level-2 slope for treatment

34

To what extent does teacher implementation of the math curriculum influence students’ math content knowledge?

slide-35
SLIDE 35

(level-1) 𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + 𝑠 π‘—π‘˜

(level -2) 𝛾0π‘˜= 𝛿00 + 𝛿01 (π‘ˆπ‘¦)1π‘˜ + 𝑣0π‘˜

(π‘ˆπ‘¦)1π‘˜ = value on the level-2 predictor (π‘ˆπ‘¦)1π‘˜= value for treatment (0 = no treatment, 1 = treatment)*

*dummy coded

35

To what extent does teacher implementation of the math curriculum influence students’ math content knowledge?

slide-36
SLIDE 36

(level-1) 𝑍

π‘—π‘˜= 𝛾0π‘˜ + 𝛾1π‘˜ (πΊπ‘“π‘›π‘π‘šπ‘“π‘—π‘˜) + 𝑠 π‘—π‘˜

(level -2) 𝛾0π‘˜= 𝛿00 + 𝛿01 (π‘ˆπ‘¦)1π‘˜ + 𝑣0π‘˜

𝑣0π‘˜ = random effects of the π‘˜th level-2 unit adjusted for treatment on the intercept 𝑣0π‘˜ = unique effect for each school on mean math achievement

36

To what extent does teacher implementation of the math curriculum influence students’ math content knowledge?

slide-37
SLIDE 37

37

HLM Challenges

x

slide-38
SLIDE 38

Insufficient power at level -1 or level-2

38

HLM Challenges: Power

slide-39
SLIDE 39

Measures need strong psychometric properties

39

HLM challenges: Meeting model assumptions

slide-40
SLIDE 40

Level-1 residuals need to be independent and normally distributed

40

HLM challenges: Meeting model assumptions

slide-41
SLIDE 41

41

Questi tion

  • ns?

s?

slide-42
SLIDE 42

42

What is HLM? When to use HLM Example HLM Use Pro Tips: What (not) to do

Webinar Sections

Tools & Resources

slide-43
SLIDE 43

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

NO NO YES YES YES NO

When to use HLM

YES NO

slide-44
SLIDE 44

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

NO NO NO YES YES YES NO

When to use HLM

YES NO

slide-45
SLIDE 45

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

YES NO

slide-46
SLIDE 46

Does the data have a nested structure? Is there sufficient power at the highest level?

Is there an adequate ICC to warrant multi- level modelling? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations

Consider regression or another more appropriate design.

YES NO NO NO NO YES YES YES YES

When to use HLM Are you familiar with Power Analysis? Are you familiar with Optimal Design?

slide-47
SLIDE 47

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

YES NO

slide-48
SLIDE 48

Does the data have a nested structure? Is there sufficient power at the highest level?

Is there an adequate ICC to warrant multi- level modelling? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations

Consider regression or another more appropriate design.

YES NO NO NO YES YES NO

Bonus!

WWC Recommends clustering adjustment for single-level analyses with multiple levels for significant findings.

  • 1. Compute test statistic for effect size
  • 2. Adjust test statistic and degrees of freedom for effect size
  • 3. Identify significance value

Handy Resource: http://www.air.org/resource/wwc-phase-i-computation-tools-4-15-10 When to use HLM

slide-49
SLIDE 49

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

NO

slide-50
SLIDE 50

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

NO

slide-51
SLIDE 51

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

NO YES

slide-52
SLIDE 52

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

NO YES

slide-53
SLIDE 53

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

NO YES

slide-54
SLIDE 54

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES NO

When to use HLM

NO YES

slide-55
SLIDE 55

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES NO

When to use HLM

NO YES YES

slide-56
SLIDE 56

To what extent does teacher participation in the MSP contribute to student science content knowledge?

Is HLM appropr

  • pria

iate? e?

57

When to use HLM

Sce cena nario io

You are the evaluator of an MSP designed to train teams of teachers in science content knowledge for 8th graders. Why or Why Not

  • t?

? Major activities include an intensive summer institute, learning teams of involved teachers, teacher leaders, and research activities. Teachers randomly assigned to training or not (cluster randomized trial) N teachers = 17 Tx, 42 Control N students = 2,025

slide-57
SLIDE 57

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

YES NO

slide-58
SLIDE 58

Does the data have a nested structure? Is there sufficient power at the highest level? Does your ICC reach an acceptable level? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression or another more appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

YES NO

slide-59
SLIDE 59

60

When to use HLM: Power Raudenbush, S. W., et al. (2011). Optimal Design Software for Multi-level and Longitudinal Research (Version 3.01) [Software]. Available from www.wtg wtgran antf tfoun

  • undation.or

dation.org. . http://sit sitema emaker er.umi umich. ch.edu/gr edu/group up-based/opti ased/optima mal_design _design_s _sof

  • ftw

tware are

slide-60
SLIDE 60

Does the data have a nested structure? Is there sufficient power at the highest level? What is the ICC? Are assumptions met? Is there sufficient power at the lowest level? HLM is likely a good choice Consider HLM with reservations Consider regression

  • r another more

appropriate design.

YES NO NO NO YES YES YES NO

When to use HLM

Keep in mind as you move forward with analysis planning.

slide-61
SLIDE 61

Decision cision OLS Regression was used to analyze the data due to insufficient power to detect an effect of the program.

63

When to use HLM (or not)

Do you agr gree ee? Wh Why or Wh Why not?

slide-62
SLIDE 62

64

Questi tion

  • ns?

s?

slide-63
SLIDE 63

65

What is HLM? When to use HLM Example HLM Use Pro Tips: What (not) to do

Webinar Sections

Tools & Resources

slide-64
SLIDE 64

66

When to use HLM

Scena cenario io: Reading HLM Reports

You are the evaluator of an MSP designed to train teams of teachers in science content knowledge for 8th graders. Major activities include an intensive summer institute, learning teams of involved teachers, teacher leaders, and research activities. N teacher hers s = 148 Tx Tx, 150 Contr trol

  • l*

Teachers randomly assigned to training or not (cluster randomized trial) N students = 2,358** To what extent does teacher participation in the MSP contribute to student science content knowledge (assuming all students have scores for the standardized state science test)? You are developing the analysis plan for this project.

* Teac eacher her level l va varia iable bles: s: MSP teac acher her, , MSP Leade ader **Stude dent t level l covar ariat iates: es: Gende der, , Title tle I status, , Indiv ividualiz idualized d Educatio ion Plan (IEP), ), Hispan anic, ic, English lish Language age Lear arner er, , prio ior Norma mal l Curve Equ quiv ivalent lent scor

  • re

e (NCE) CE)

slide-65
SLIDE 65

67

Example HLM Use

Model 1 Model 2 Model 3 Model 4 Est. SE Est. SE Est. SE Est. SE Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560 Gender 1.323 0.731 1.323 0.731 1.283 0.731 Title I

  • 0.956

1.972

  • 0.956

1.973

  • 0.904

2.010 IEP

  • 17.525***

1.250

  • 17.524***

1.251

  • 17.505***

1.250 Hispanic

  • 10.348***

1.250

  • 10.348***

1.250

  • 10.287***

1.250 ELL

  • 6.659***

1.236

  • 6.659***

1.236

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.244 0.124 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3 Intraclass Correlation (ICC) .077

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

*p < .05, **p < .01, ***p < .001

The first st model el is always ys a null model el.

slide-66
SLIDE 66

68

Example HLM Use

Model 1 Model 2 Model 3 Model 4 Est. SE Est. SE Est. SE Est. SE Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560 Gender 1.323 0.731 1.323 0.731 1.283 0.731 Title I

  • 0.956

1.972

  • 0.956

1.973

  • 0.904

2.010 IEP

  • 17.525***

1.250

  • 17.524***

1.251

  • 17.505***

1.250 Hispanic

  • 10.348***

1.250

  • 10.348***

1.250

  • 10.287***

1.250 ELL

  • 6.659***

1.236

  • 6.659***

1.236

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.244 0.124 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3 Intraclass Correlation (ICC) .077

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

*p < .05, **p < .01, ***p < .001

On average, erage, participant icipants s had a NCE score e of 52.004, 4, with a standa ndard d error of 1. 1.660. 0.

slide-67
SLIDE 67

69

Example HLM Use

Model 1 Model 2 Model 3 Model 4 Est. SE Est. SE Est. SE Est. SE Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560 Gender 1.323 0.731 1.323 0.731 1.283 0.731 Title I

  • 0.956

1.972

  • 0.956

1.973

  • 0.904

2.010 IEP

  • 17.525***

1.250

  • 17.524***

1.251

  • 17.505***

1.250 Hispanic

  • 10.348***

1.250

  • 10.348***

1.250

  • 10.287***

1.250 ELL

  • 6.659***

1.236

  • 6.659***

1.236

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.244 0.124 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3 Intraclass Correlation (ICC) .077

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

*p < .05, **p < .01, ***p < .001

Deviance iance indicat cates es model el fit, and lowe wer devian ance ce indicat cates es bett etter er fit.

slide-68
SLIDE 68

70

Example HLM Use

Model 1 Model 2 Model 3 Model 4 Est. SE Est. SE Est. SE Est. SE Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560 Gender 1.323 0.731 1.323 0.731 1.283 0.731 Title I

  • 0.956

1.972

  • 0.956

1.973

  • 0.904

2.010 IEP

  • 17.525***

1.250

  • 17.524***

1.251

  • 17.505***

1.250 Hispanic

  • 10.348***

1.250

  • 10.348***

1.250

  • 10.287***

1.250 ELL

  • 6.659***

1.236

  • 6.659***

1.236

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.244 0.124 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3 Intraclass Correlation (ICC) .077

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

*p < .05, **p < .01, ***p < .001

7.7% % of the varian ance ce in scien ence ce achieveme ement nt was due to variat atio ion n bet etween en teacher hers. s.

slide-69
SLIDE 69

71

Example HLM Use

Model 1 Model 2 Model 3 Model 4 Est. SE Est. SE Est. SE Est. SE Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560 Gender 1.323 0.731 1.323 0.731 1.283 0.731 Title I

  • 0.956

1.972

  • 0.956

1.973

  • 0.904

2.010 IEP

  • 17.525***

1.250

  • 17.524***

1.251

  • 17.505***

1.250 Hispanic

  • 10.348***

1.250

  • 10.348***

1.250

  • 10.287***

1.250 ELL

  • 6.659***

1.236

  • 6.659***

1.236

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.244 0.124 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3 Intraclass Correlation (ICC) .077

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

Model el 2 typically cally adds s Level 1 predict ctor

  • rs
slide-70
SLIDE 70

72

Example HLM Use

Model 1 Model 2 Model 3 Model 4 Est. SE Est. SE Est. SE Est. SE Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560 Gender 1.323 0.731 1.323 0.731 1.283 0.731 Title I

  • 0.956

1.972

  • 0.956

1.973

  • 0.904

2.010 IEP

  • 17.525***

1.250

  • 17.524***

1.251

  • 17.505***

1.250 Hispanic

  • 10.348***

1.250

  • 10.348***

1.250

  • 10.287***

1.250 ELL

  • 6.659***

1.236

  • 6.659***

1.236

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.244 0.124 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3 Intraclass Correlation (ICC) .077

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

Model el 3 typical cally y adds s predict ctor

  • rs

s of interest est

slide-71
SLIDE 71

73

Example HLM Use

Model 1 Model 2 Model 3 Model 4 Est. SE Est. SE Est. SE Est. SE Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560 Gender 1.323 0.731 1.323 0.731 1.283 0.731 Title I

  • 0.956

1.972

  • 0.956

1.973

  • 0.904

2.010 IEP

  • 17.525***

1.250

  • 17.524***

1.251

  • 17.505***

1.250 Hispanic

  • 10.348***

1.250

  • 10.348***

1.250

  • 10.287***

1.250 ELL

  • 6.659***

1.236

  • 6.659***

1.236

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.244 0.124 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3 Intraclass Correlation (ICC) .077

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

Deviance iance decrea ease sed d from Model el 1 to Model 4.

slide-72
SLIDE 72

74

Example HLM Use

Model 4 Est. SE Intercept 52.214 0.560 Gender 1.283 0.731 Title I

  • 0.904

2.010 IEP

  • 17.505***

1.250 Hispanic

  • 10.287***

1.250 ELL

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,330.3 Intraclass Correlation (ICC)

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

Model el 4 is the final al model el.

slide-73
SLIDE 73

75

Example HLM Use

Model 4 Est. SE Intercept 52.214 0.560 Gender 1.283 0.731 Title I

  • 0.904

2.010 IEP

  • 17.505***

1.250 Hispanic

  • 10.287***

1.250 ELL

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,330.3 Intraclass Correlation (ICC)

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

On average, erage, student dents s scor

  • red

d 52.214 4 NCE units. ts.

slide-74
SLIDE 74

78

Example HLM Use

Model 4 Est. SE Intercept 52.214 0.560 Gender 1.283 0.731 Title I

  • 0.904

2.010 IEP

  • 17.505***

1.250 Hispanic

  • 10.287***

1.250 ELL

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,330.3 Intraclass Correlation (ICC)

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

On average, erage, student dents s with h an IEP scored ed 17.50 505 5 points ts lower than those se without hout.

slide-75
SLIDE 75

79

Example HLM Use

Model 4 Est. SE Intercept 52.214 0.560 Gender 1.283 0.731 Title I

  • 0.904

2.010 IEP

  • 17.505***

1.250 Hispanic

  • 10.287***

1.250 ELL

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,330.3 Intraclass Correlation (ICC)

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

On average, erage, Hispani nic c studen ents ts scored ed 10.287 7 points ts lower r than non-Hisp Hispani nic c student dents. s.

slide-76
SLIDE 76

80

Example HLM Use

Model 4 Est. SE Intercept 52.214 0.560 Gender 1.283 0.731 Title I

  • 0.904

2.010 IEP

  • 17.505***

1.250 Hispanic

  • 10.287***

1.250 ELL

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,330.3 Intraclass Correlation (ICC)

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

On average, erage, ELL student dents s scored ed 6.686 points ts lower than non-ELL student dents. s.

slide-77
SLIDE 77

81

Example HLM Use

Model 4 Est. SE Intercept 52.214 0.560 Gender 1.283 0.731 Title I

  • 0.904

2.010 IEP

  • 17.505***

1.250 Hispanic

  • 10.287***

1.250 ELL

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,330.3 Intraclass Correlation (ICC)

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

For every NCE unit scor

  • re on the pre-test

est, student dents s gaine ned 1. 1.168 NCE units ts on the post-test est, on average. age.

slide-78
SLIDE 78

82

Example HLM Use

Model 4 Est. SE Intercept 52.214 0.560 Gender 1.283 0.731 Title I

  • 0.904

2.010 IEP

  • 17.505***

1.250 Hispanic

  • 10.287***

1.250 ELL

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,330.3 Intraclass Correlation (ICC)

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

Studen ents ts who had an MSP Teacher her scor

  • red

ed 2.522 points ts higher er than n those se who did not

  • t.

.

slide-79
SLIDE 79

83

Example HLM Use

Model 4 Est. SE Intercept 52.214 0.560 Gender 1.283 0.731 Title I

  • 0.904

2.010 IEP

  • 17.505***

1.250 Hispanic

  • 10.287***

1.250 ELL

  • 6.686***

1.235 Normal Curve Equivalent (NCE) pretest 1.168*** 0.112 MSP Teacher 2.522* 0.897 MSP Leader 1.045 1.349 HLM Deviance 15,330.3 Intraclass Correlation (ICC)

*p < .05, **p < .01, ***p < .001

Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)

There e was no difference erence in scores es for student dents s in class sses es that t were taught ht by MSP Leaders s compared ared to those se who were not

  • t.
slide-80
SLIDE 80

84

Questi tion

  • ns?

s?

slide-81
SLIDE 81

85

What is HLM? When to use HLM Example HLM Use Pro Tips: What (not) to do

Webinar Sections

Tools & Resources

slide-82
SLIDE 82

86

Pro Tips: What (not) to do

Make sure all variables are dummy coded appropriately, with 1/0 for each category and a reference group.

Race X1 X1 X2 X2 X3 X3 X4 X4 White 1 Black 1 Hispanic 1 Other

slide-83
SLIDE 83

87

Think strategically about centering your variables.

Un Uncen centere ered: : Xij

ij

Group up-mean ean cent ntered: red: Xij

ij βˆ’ 𝒀

j Grand nd-mean ean cent ntere ered: d: Xij

ij βˆ’ 𝒀

Β·Β·

Β·Β·

Pro Tips: What (not) to do

slide-84
SLIDE 84

88

Strategically build your model.

Nu Null Model Covariat iates es Level 1 Predict ctors s

  • f Interest

rest Level 2 Predict ctors s

  • f Interest

rest

Pro Tips: What (not) to do

slide-85
SLIDE 85

89

Make sure to report relevant statistics. According to Abt Associates’ Guide for rigorous MSP evaluations and reporting:

Pro Tips: What (not) to do

slide-86
SLIDE 86

90

What is a challenge you have faced when running HLM or considering whether or not to use HLM?

Pro Tips: What (not) to do

slide-87
SLIDE 87

91

What is HLM? When to use HLM Example HLM Use Pro Tips: What (not) to do

Webinar Sections

Tools & Resources

slide-88
SLIDE 88

Software Options for HLM Analysis

92 SPSS Tools and Resources

slide-89
SLIDE 89

93

Tools and Resources

Resour urces ces



SSI Website to download HLM and find resources: http://www.ssicentral.com/hlm/resources.html



Schochet, P. Z., Puma, M., & Deke, J. (2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE 2014–4017). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and

  • Development. Retrieved from http://ies.ed.gov/ncee/edlabs.



Raudenbush, S. W., et al. (2011). Optimal Design Software for Multi-level and Longitudinal Research (Version 3.01) [Software]. Available from www.wtgrantfoundation.org or //sitemaker.umich.edu/group-based/optimal_design_software.



Variance Almanac of Academic Achievement : https://arc.uchicago.edu/reese/variance- almanac-academic-achievement

slide-90
SLIDE 90

94

Questi tion

  • ns?

s?

slide-91
SLIDE 91

95

TEAMS Resources & Tools

TEAMS MS MSP P Project ject Document ment Self-Ap Apprai praisal

  • Purpos

rpose of the Evaluation ation

  • Evaluatio

ation n Design & Measureme rement nt

  • Analys

lysis is

  • Generaliz

lizab ability ility, , Represent ntativ ativenes ness, s, Utility ity http:/ p://t /teams. ams.mspnet.o mspnet.org rg/ / index. x.cfm fm/2 /27152 52

slide-92
SLIDE 92

Contact Us!

Kar Karen Drill kdrill@r @rmccorp mccorp.com com Emma ma Espel el espel pel@r @rmcr mcres.com es.com RMC Research Corporation 111 SW Columbia St., Suite 1030 Portland, OR 98201-5883 RMC Research Corporation 633 17th St., Suite 2100 Denver, CO 80202-1620

96

John John T. Sutton

  • n,

, PI sutt tton@ n@rmcr mcres.com es.com Dave e Weaver er, , Co-PI PI dwea eaver er@r @rmccorp mccorp.com com RMC Research Corporation 633 17th St., Suite 2100 Denver, CO 80202-1620 Phone: 303-825-3636 Toll Free: 800-922-3636 Fax: 303-825-1626 RMC Research Corporation 111 SW Columbia St., Suite 1030 Portland, OR 97201-5883 Phone: 503-223-8248 Toll Free: 800-788-1887 Fax: 503-223-8399

slide-93
SLIDE 93

97

Follow Up Survey

Please Complete the Feedback Survey

URL: http://www.rmccorp.com/TEAMS/ Username: 23569 Password: TEAMS Thank hank You!

NSF # DRL 1238120