Innovations for Testing Multilevel Causal Theory in Implementation - - PowerPoint PPT Presentation

▶
innovations for testing multilevel
SMART_READER_LITE
LIVE PREVIEW

Innovations for Testing Multilevel Causal Theory in Implementation - - PowerPoint PPT Presentation

Mar 05, 2024 275 likes •456 views

Methodological Challenges and Innovations for Testing Multilevel Causal Theory in Implementation Science Nate Williams, PhD Steve Marcus, PhD Boise State University University of Pennsylvania School of Social Work School of Social Policy

slide-1
SLIDE 1

WWW.UPENN.EDU

Methodological Challenges and Innovations for Testing Multilevel Causal Theory in Implementation Science

1

Nate Williams, PhD Boise State University School of Social Work Steve Marcus, PhD University of Pennsylvania School of Social Policy & Practice

slide-2
SLIDE 2

WWW.UPENN.EDU

2

X Y

c

M

a b c'

X Y

Mediation Analysis

Does variation in X explain variation in M? Does variation in M explain variation in Y?

slide-3
SLIDE 3

WWW.UPENN.EDU

Tests for Mediation

Test Pros Cons Baron & Kenny causal steps approach

  • Relative ease of implementation
  • Familiar to reviewers
  • Low power
  • Doesn’t estimate or test

parameter of interest

  • No effect size

Joint significance test

  • Good power
  • Ease of implementation
  • NHST approach
  • Doesn’t estimate or test the

parameter of interest

  • No effect size

Product of coefficients approach with Sobel test

  • Ease of implementation
  • Can produce an effect size
  • Familiar to reviewers
  • Suboptimal power
  • Inaccurate normality assumption

Product of coefficients approach with asymmetric 95% CIs

  • Optimal power
  • Ease of implementation
  • Excellent Type I error protection
  • Can produce an effect size
  • Directly tests parameter of interest
  • Doesn’t provide p-value
  • Not an NHST approach

3

 

(Hayes & Scharkow, 2013; MacKinnon et al., 2001, MacKinnon et al., 2004)

slide-4
SLIDE 4

WWW.UPENN.EDU

Product of Coefficients Approach

  • 1. Run a series of regression models

that estimate the paths of interest

  • 2. Multiply coefficients to obtain an

estimate of the indirect effect(s)

  • 3. Test the statistical significance of the

indirect effect(s) using the Joint Significance Test and develop Asymmetric 95% Confidence Intervals

  • 4. Generate measures of effect size

4

M

a b c'

X Y

a*b = ??? Asymmetric 95% CI = ?? to ?? Joint Significance Test = ?? Pm = ??

slide-5
SLIDE 5

WWW.UPENN.EDU

Multilevel Data Creates Problems for Standard Mediation Analysis

(Krull & MacKinnon, 2001; Preacher, 2015) Agencies Clinicians Patients

1. Nesting creates non-independence of

  • bservations which violates the

assumptions of standard statistical models

slide-6
SLIDE 6

WWW.UPENN.EDU

Multilevel Theory Creates Problems for Standard Mediation Analysis

6

Intentions Self-Efficacy Norms Attitudes Skill Beliefs Intensity Knowledge School District State Federal Implementation Leadership and Climate

2. Multilevel theory is not easily incorporated into standard statistical models

  • X, M, & Y might

reside at different levels

  • Variation in the
  • utcome is assumed

to occur at multiple levels

slide-7
SLIDE 7

WWW.UPENN.EDU

Multilevel Modeling as a Partial Solution

  • MLM allows investigators to treat variation in the outcome that occurs across clusters as a

phenomena of interest rather than a nuisance

  • MLM permits the inclusion of predictor variables at multiple levels
  • Relative to a fixed effects approach, MLM allows researchers to account for the non-

independence of nested observations in a way that is both parsimonious and flexible

7

slide-8
SLIDE 8

WWW.UPENN.EDU

Using MLM to Conduct Mediation Analyses

  • The product of coefficients approach can be easily extended to multilevel mediation models

(Krull & MacKinnon, 2001; Zhang et al., 2009)

8

Organizational antecedent Organizational Mediator Clinician

  • utcome

a b c'

Level 2 (Organization) Level 1 (Individual)

Clinician mediator Clinician

  • utcome

a b c'

Level 2 (Organization) Level 1 (Individual)

Organizational antecedent

slide-9
SLIDE 9

WWW.UPENN.EDU

  • Level 1 variables have variance at level 1 (within groups) and level 2 (between groups)

9

The Problem of Conflated Slopes in Multilevel Mediation Analysis

slide-10
SLIDE 10

WWW.UPENN.EDU

10

  • Because a level 1 X and a level 1 Y have unique variances at Level 1 and Level 2, their relationship to

each other can differ across levels

Two Potentially Different Relationships between X and Y

  • Estimating the X – Y relationship using a single slope, as is sometimes recommended for multilevel

mediation analyses, “conflates” the two relationships and leads to biased parameter estimates and incorrect statistical tests in mediation analysis (Zhang et al., 2009; Enders & Tofighi, 2007; Kreft et al., 1995)

slide-11
SLIDE 11

WWW.UPENN.EDU

  • Two new variables are created which partition the

variance in M into two parts:

  • group means (𝑁𝒌)
  • group-mean centered scores 𝑁𝑗𝑘 − 𝑁𝒌
  • Both variables are used in the analysis but only
  • ne variable (group means) is used to calculate

the indirect effect

11

Un-conflating Slopes in Multilevel Mediation Analysis

Centered within Context with Means Reintroduced Approach– CWCM (Zhang et al., 2009)

slide-12
SLIDE 12

WWW.UPENN.EDU

12

Example of Un-conflated MLM Mediation Analysis

Organizational culture Clinician intentions Clinician behavior (use of EBP) a b c'

a*b = .035 Asymmetric 95% CI = .011 to .067 Joint Significance Test = Sig. Pm = .62

slide-13
SLIDE 13

WWW.UPENN.EDU

Conclusions

  • Standard analytic approaches to mediation analysis are often insufficient for hypothesis

testing in implementation science because of nested data and multilevel theory

  • Multilevel modeling is a useful tool for addressing these issues but problems can arise if

investigators ignore the “conflated slopes issue”

  • Simple procedures are available that allow investigators to un-conflate slopes in MLM and

generate accurate estimates of indirect effects

13

slide-14
SLIDE 14

WWW.UPENN.EDU

THANK YOU!

Nate Williams, PhD natewilliams@boisestate.edu Steve Marcus, PhD marcuss@upenn.edu

14

slide-15
SLIDE 15

WWW.UPENN.EDU

References

15

Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. Psychological Methods, 12(2), 121-138. Hayes, A. F., & Scharkow, M. (2013). The Relative Trustworthiness of Inferential Tests of the Indirect Effect in Statistical Mediation Analysis: Does Method Really Matter?. Psychological Science (0956- 7976), 24(10), 1918-1927. Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995). The effects of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1-21. Krull, J. L., & MacKinnon, D. P. (2001). Multilevel Modeling of Individual and Group Level Mediated

  • Effects. Multivariate Behavioral Research, 36(2), 249-277. doi:10.1207/S15327906MBR3602_06

MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison

  • f methods to test mediation and other intervening variable effects. Psychological Methods, 7(1), 83-

104.

slide-16
SLIDE 16

WWW.UPENN.EDU

References

16

Mackinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods. Multivariate Behavioral Research, 39(1), 99. Preacher, Kristopher J. 2015. Advances in mediation analysis: a survey and synthesis of new developments." Annual Review Of Psychology 66, 825-852. Preacher, K. J., & Kelley, K. (2011). Effect size measures for mediation models: quantitative strategies for communicating indirect effects. Psychological Methods, 16(2), 93-115. doi:10.1037/a0022658 Zhang, Z., Zyphur, M. J., & Preacher, K. J. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12(4), 695-719. doi:10.1177/1094428108327450