What is an Adaptive Implementation Intervention? Why do we need - - PowerPoint PPT Presentation

What is an Adaptive Implementation Intervention? Why do we need them? How do we optimize them?

Daniel Almirall, Amy Kilbourne, Andrew Quanbeck, Shawna Smith and Many Friends Survey Research Center, Institute for Social Research Department of Statistics, LS&A University of Michigan 5 December 2019 D&I Academy Health Arlington, VA

D&I Adaptive Implementation Interventions Dec 2019 1 / 118

My Collaborators

Methods/Statistics Collaborators Inbal Nahum-Shani, Mich Susan A. Murphy, Harvard Linda M. Collins, PSU Daniel F. McCaffrey, ETS Domain Science Collaborators Connie Kasari, UCLA Amy Kilbourne, Mich Andrew Quanbeck, Wisc Shawna Smith, Mich Megan Patrick, UMN/ITR Meredith G-S, UMN/ITR Ahnalee Brincks, MSU Statistics Students Nick Seewald, PhD Brook Luers, PhD Tim NeCamp, PhD Kelly Speth, PhD Olivia Hackworth, PhD Madison Stoms, Ugrad Sophia Luo, Ugrad

D&I Adaptive Implementation Interventions Dec 2019 2 / 118

Don’t worry... I don’t really have 100+ slides for a 8min talk!

D&I Adaptive Implementation Interventions Dec 2019 3 / 118

Outline

Two challenges we face in D&I

◮ Challenge#1: One size does not fit all: different strategies needed ◮ Challenge#2: Many implementation strategies, how to combine them

Adaptive Implementation Interventions Developing an optimized Adaptive Implementation Intervention Two Example Trials

◮ ASIC: Adaptive Implementation of CBT in Schools (PI: Kilbourne) ◮ Opioid Prescribing in Primary Care (PI: Quanbeck) D&I Adaptive Implementation Interventions Dec 2019 4 / 118

Challenge #1: One size does not fit all organizations (or providers)

D&I Adaptive Implementation Interventions Dec 2019 5 / 118

Organization-level heterogeneity in response to implementation intervention

Implementation strategies that work for one organization may not work for the other (between-organization heterogeneity) Implementation strategies that work now, may not work as well later and vice-versa (within-organization heterogeneity) Sequencing of the implementation strategies matters Cost of the implementation strategies matters

D&I Adaptive Implementation Interventions Dec 2019 6 / 118

Possible solution is a sequential, organization-specific approach whereby strategies are tailored over time to the needs of the organization

Consequential decisions about strategies are being made over time Requires decisions to be replicable Requires a decision-maker (e.g., the Implementer)

D&I Adaptive Implementation Interventions Dec 2019 7 / 118

Possible solution is a sequential, organization-specific approach whereby strategies are ✘✘✘✘

✘

tailored matched over time to the needs of the organization

Consequential decisions about strategies are being made over time Requires decisions to be replicable Requires a decision-maker (e.g., the Implementer)

D&I Adaptive Implementation Interventions Dec 2019 8 / 118

Possible solution is a sequential, organization-specific approach whereby strategies are ✘✘✘✘

✘

tailored ✘✘✘✘✘

✘

matched adapted

• ver time to the needs of the organization

Consequential decisions about strategies are being made over time Requires decisions to be replicable Requires a decision-maker (e.g., the Implementer) Adaptive Implementation Interventions provide a guide for intervening this way.

D&I Adaptive Implementation Interventions Dec 2019 9 / 118

What is an Adaptive Implementation Intervention?

D&I Adaptive Implementation Interventions Dec 2019 10 / 118

Definition of an Adaptive Implementation Intervention

A pre-specified, replicable sequence of decision rules used to guide whether, how, or when to alter the implementation strategy (e.g., change the monitoring schedule, package of strategies, duration, frequency or amount) at critical decision points during the implementation of evidence-based practices.

D&I Adaptive Implementation Interventions Dec 2019 11 / 118

Example of an Adaptive Implementation Intervention

To improve the adoption of CBT in high-schools across Michigan (PI: Kilbourne, Co-I: Smith, Methodologist: Almirall)

D&I Adaptive Implementation Interventions Dec 2019 12 / 118

What are the Components of an Adaptive Implementation Intervention?

Critical implementation decision points Implementation strategies Tailoring variables Decision rules Proximal and distal goals that guide the above components

D&I Adaptive Implementation Interventions Dec 2019 13 / 118

Who are the Adaptive Implementation Interventions for?

AIIs are intended to guide implementation decision-makers Implementation practitioners Community service providers Service provider associations Policy makers Director: Elizabeth Koschmann

D&I Adaptive Implementation Interventions Dec 2019 14 / 118

Example of an Adaptive Implementation Intervention

To improve the adoption of CBT in high-schools across Michigan (PI: Kilbourne, Co-I: Smith, Methodologist: Almirall)

D&I Adaptive Implementation Interventions Dec 2019 15 / 118

An Adaptive Implementation Intervention is not a research method

It is not a type of experimental design It is not an adaptive trial design It is not a method to adapt the EBP It is not a way of conducting pilot or usability studies An Adaptive Implementation Intervention is a package of implementation strategies that lives in the real-world.

D&I Adaptive Implementation Interventions Dec 2019 16 / 118

Challenge #2: So many implementation strategies; So many decisions; How do we combine the strategies to make a good Adaptive Implementation Intervention?

D&I Adaptive Implementation Interventions Dec 2019 17 / 118

What does it mean develop an optimized Adaptive Implementation Intervention?

D&I Adaptive Implementation Interventions Dec 2019 18 / 118

A MOST useful way to Think about Your Research on Adaptive Implementation Interventions

Multi-phase Optimization STrategy (MOST) Framework: Collins (2018); Collins & Kugler (2018); Almirall, et al. (2018)

D&I Adaptive Implementation Interventions Dec 2019 19 / 118

Example of an Adaptive Implementation Intervention

To improve the adoption of CBT in high-schools across Michigan (PI: Kilbourne, Co-I: Smith, Methodologist: Almirall)

D&I Adaptive Implementation Interventions Dec 2019 20 / 118

Example Scientific Questions that are Critical when Optimizing an Adaptive Implementation Intervention

What is the effect of offering vs not offering CBT skills coaching at Month 2? Which are the schools that benefit the most from CBT skills coaching? And which are the schools for which there is no evidence of an effect of CBT skills coaching? For schools that are not yet adopting CBT by month 4: What is the effect of augmenting implementation with Facilitation? And which are the schools that benefit the most from augmenting with Facilitation?

D&I Adaptive Implementation Interventions Dec 2019 21 / 118

Insufficient Empirical Evidence, Expertise, Conceptual Models or Theories to Address such Questions

Often, it is insufficient or not optimal to Expert implementation practitioner opinion Other clinical expertise Piece together an Adaptive Implementation Intevrention using results from separate Hybrid Trial Type-3 studies

D&I Adaptive Implementation Interventions Dec 2019 22 / 118

Clustered Sequential Multiple Assignment Randomized Trials

D&I Adaptive Implementation Interventions Dec 2019 23 / 118

What is a Clustered Sequential Multiple Assignment Randomized Trial (SMART)?

Type of multi-stage, randomized trial design—a factorial design. One type of trial you might use for optimization At each stage, organizations/providers are randomized to a set of feasible/ethical implementation strategies. Clustered: strategies at organization level (e.g., clinic) and primary

• utcome at level within organization (e.g., providers or patient)

SMARTs were developed explicitly for the purpose of empirically developing optimized Adaptive (Implementation) Interventions.

D&I Adaptive Implementation Interventions Dec 2019 24 / 118

Prototypical SMART

SMART Study Example 1: Improving Opioid Prescribing in Primary Care

D&I Adaptive Implementation Interventions Dec 2019 26 / 118

Improving Opioid Prescribing in Primary Care

PI: Quanbeck, Methodologist: Almirall

Improving Opioid Prescribing in Primary Care

Primary Aim: to compare the effect of the sequence of implementation strategies (EM+AF, AF+PF, AF+PF+PPC) versus EM+AF alone on avg provider-level morphine milligram equivalent dose over 18 months. Emprically Develop an Optimized AII: (a) Pre-specified Candidate Moderators: (i) Does existence of opioid policy, experience doing QI at clinic-level, or size of clinic moderate the effect of adding PF at month 3? (ii) Does number of high-dose opioid patients (measured at baseline and months 3 and 6) moderate the effect

• f adding PPC at month 6?

(b) To qualitatively assess other contextual factors that may influence the effectiveness of different implementation strategies at different stages, and to empirically test these factors.

SMART Study Example 2: Adaptive School-based Implementation for CBT

D&I Adaptive Implementation Interventions Dec 2019 29 / 118

Adaptive School-based Implementation of CBT

PI: Kilbourne, Co-I: Smith, Methodologist: Almirall

Adaptive School-based Implementation of CBT

Primary Aim: Compare the AII (w/Coaching+Facilitation) versus REP alone on number of SP-delivered CBT sessions delivered over 18mos. versus

Adaptive School-based Implementation of CBT

Emprically Develop an More Optimized AII: (a) Is the effect of augmenting REP with Coaching moderated by school-aggregated school provider training or baseline perceptions of CBT? (b) Among schools that show a potential need for further support, is the effect of augmentation with Facilitation moderated by school-level CBT delivery during first 8 weeks post-randomization, number of barriers to CBT reported 8 weeks post-randomization, satisfaction with current implementation support, or school administrator support for adoption of innovation?

Adaptive School-based Implementation of CBT

PI: Kilbourne, Co-I: Smith, Methodologist: Almirall

What did you learn in the last 8 minutes?

You learned about Adaptive Implementation Interventions You learned about SMART designs for empirically developing an

• ptimized Adaptive Implementation Intervention

You learned about 2 example SMARTs

Thank you! Questions?

dalmiral@umich.edu, http://www-personal.umich.edu/∼dalmiral/ If you are an Education researcher: http://d3lab-isr.com/training

D&I Adaptive Implementation Interventions Dec 2019 35 / 118

References about AIIs and Clustered SMARTs

Kilbourne, Smith ... Almirall (2018). Adaptive School-based Implementation

• f CBT (ASIC): clustered-SMART for building an optimized adaptive

implementation intervention to improve uptake of mental health interventions in schools., Implementation Science. NeCamp, Kilbourne, Almirall (2017). Cluster-level adaptive interventions and sequential, multiple assignment, randomized trials: Estimation and sample size considerations, Statistical Methods in Medical Research. Kilbourne, Almirall, et al. (2014). Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disorders program, Implementation Science.

D&I Adaptive Implementation Interventions Dec 2019 36 / 118

More General References

Lu, Nahum-Shani, Kasari, Lynch, Oslin, Pelham, Fabiano, Almirall. (2016). Comparing DTRs using repeated-measures outcomes: modeling considerations in SMART studies, Stats in Medicine. Almirall, DiStefano, Chang, Shire, Lu, Nahum-Shani, Kasari, C. (2016). Adaptive interventions and longitudinal outcomes in minimally verbal children with ASD: Role of speech-generating devices, JCCAP. Dziak, Yap, Almirall, McKay, Lynch, and Nahum-Shani (2019). A Data Analysis Method for Using Longitudinal Binary Outcome Data from a SMART to Compare Adaptive Interventions. MBR, 1-24. Nahum-Shani, Almirall, Yap, McKay, Lynch, Freiheit, Dziak (2019). SMART Longitudinal Analysis: A Tutorial for Using Repeated Outcome Measures from SMART Studies to Compare Adaptive Interventions. Psych Methods Seewald, Nahum-Shani, McKay, Almirall (under review). Sample size considerations for comparing DTRs in a sequentially-randomized trial with a continuous longitudinal outcome. Luers, Qian, Nahum-Shani, Kasari, Almirall (in progress). Longitudinal Mixed-effects Models to compare DTRs in Sequentially-randomized Trials

D&I Adaptive Implementation Interventions Dec 2019 37 / 118

Even More General References

Almirall, Kasari, McCaffrey, Nahum-Shani. (2018). Developing Optimized Adaptive Interventions in Education, Journal of Research on Educational Effectiveness. Almirall, Nahum-Shani, Wang, Kasari. (2018). Experimental Designs for Research on Adaptive Interventions: Singly- and Sequentially-Randomized Trials, Optimization of Multicomponent Behavioral Biobehavioral and Biomedical Interventions using MOST. L. Collins, K. Kugler (Editors). Almirall, Compton, Gunlicks-Stoessel, Duan, Murphy (2012). Designing a Pilot Sequential Multiple Assignment Randomized Trial for Developing an Adaptive Treatment Strategy.” Statistics in Medicine Kim, H. and Almirall (2016). A sample size calculator for SMART pilot studies, SIAM Undergraduate Research Journal, Vol. 9.

◮ Web-applet: https://methodologycenter.shinyapps.io/PilotShiny/ D&I Adaptive Implementation Interventions Dec 2019 38 / 118

END TALK

D&I Adaptive Implementation Interventions Dec 2019 39 / 118

Extra Slides Starting Here...

D&I Adaptive Implementation Interventions Dec 2019 40 / 118

Sample Size Formulae for Cluster-Randomized SMARTs

For comparing 2 embedded AIs that begin with different treatments in a prototypical SMART

N ≥

4(z1−α/2+z1−β)

2

m∗δ2

× (2 − r) × (1 + (m − 1)ρ) × (1 − α2) where r = response rate after stage 1 impl strategy m = avg number of SPs at each school ρ = inter-school correlation in outcome α = corr(baseline cluster-level covariate, outcome)

D&I Adaptive Implementation Interventions Dec 2019 41 / 118

Sample Size Formulae for Cluster-Randomized SMARTs

For comparing 2 embedded AIs that begin with different treatments in a prototypical SMART

N ≥

4(z1−α/2+z1−β)

2

m∗δ2

× (2 − r) × (1 + (m − 1)ρ) × (1 − α2) where r = response rate after stage 1 impl strategy m = avg number of SPs at each school ρ = inter-school correlation in outcome α = corr(baseline cluster-level covariate, outcome)

r = .1 and .5, δ = .5, ρ = .03, m = 2, α = .05, N = 100, power=80%

D&I Adaptive Implementation Interventions Dec 2019 41 / 118

Myths or Misconceptions about Adaptive Interventions and SMARTs

D&I Adaptive Implementation Interventions Dec 2019 42 / 118

Myths or Misconceptions about Adaptive Interventions

Tailoring variables cannot differ based on previous intervention An adaptive intervention must recommend a single intervention component at each decision point Adaptive interventions seek to replace clinical judgement Adaptive interventions are only relevant in treatment settings Adaptive interventions are non-standard because they involve randomization

D&I Adaptive Implementation Interventions Dec 2019 43 / 118

Interventions for Minimally Verbal Children with Autism

PIs: Kasari(UCLA), Almirall(Mich), Kaiser(Vanderbilt), Smith(Rochester), Lord(Cornell)

Myths or Misconceptions about SMART Studies

All research on adaptive interventions requires a SMART SMARTs are an alternative to the RCT SMARTs require prohibitively large sample sizes All SMARTs require Multiple-Comparisons Adjustments All SMARTs must include an embedded tailoring variable All aspects of an embedded adaptive intervention must be randomized SMARTs are a form of adaptive research design SMARTs never include a control group SMARTs require a “business as usual” control group SMARTs require multiple consents SMARTs are susceptible to high levels of study drop-out

D&I Adaptive Implementation Interventions Dec 2019 45 / 118

Interventions for Minimally Verbal Children with Autism

PIs: Kasari(UCLA), Almirall(Mich), Kaiser(Vanderbilt), Smith(Rochester), Lord(Cornell)

Myths or Misconceptions about SMART Studies

All research on adaptive interventions requires a SMART SMARTs are an alternative to the RCT SMARTs require prohibitively large sample sizes All SMARTs require Multiple-Comparisons Adjustments All SMARTs must include an embedded tailoring variable All aspects of an embedded adaptive intervention must be randomized SMARTs are a form of adaptive research design SMARTs never include a control group SMARTs require a “business as usual” control group SMARTs require multiple consents SMARTs are susceptible to high levels of study drop-out

D&I Adaptive Implementation Interventions Dec 2019 47 / 118

Multi-Tiered Systems of Support

PIs: Greg Roberts and Nathan Clemens (UT Austin)

Multi-Tiered Systems of Support

PIs: Greg Roberts and Nathan Clemens (UT Austin)

Let’s call this ”Academic Adaptive Intervention.”

Multi-Tiered Systems of Support

PIs: Greg Roberts and Nathan Clemens (UT Austin)

This study investigates the role of the self-regulation component (should it be provided in stage 1, in stage 2, or at all?) in the context of the Academic Adaptive Intervention.

Multi-Tiered Systems of Support

PIs: Greg Roberts and Nathan Clemens (UT Austin)

I call this a “Seemingly-Restricted SMART”. Here, a 2x2 SMART design.

You may be wondering about sample size/power?

Prototypical SMART

Sample Size Formulae with Repeated-measures Analyses

For comparing 2 embedded AIs that begin with different treatments in a prototypical SMART; Hypothesis test is that there is no difference in means at the end of the study

N ≥

4(z1−α/2+z1−β)

2

δ2

× (2 − r) ×

• 1 − ρ2

where r = response rate after stage 1 treatment ρ = within-person correlation in outcome

D&I Adaptive Implementation Interventions Dec 2019 54 / 118

Sample Size Formulae with Repeated-measures Analyses

For comparing 2 embedded AIs that begin with different treatments in a prototypical SMART; Hypothesis test is that there is no difference in means at the end of the study

N ≥

4(z1−α/2+z1−β)

2

δ2

× (2 − r) ×

• 1 − ρ2

where r = response rate after stage 1 treatment ρ = within-person correlation in outcome

r = ρ = δ = 1/2, α = .05, need N = 142 for 80% power. Same question using a standard 2-arm trial requires N = 96.

D&I Adaptive Implementation Interventions Dec 2019 54 / 118

Sample Size Formulae for Cluster-Randomized SMARTs

For comparing 2 embedded AIs that begin with different treatments in a prototypical SMART

N ≥

4(z1−α/2+z1−β)

2

m∗δ2

× (2 − r) × (1 + (m − 1)ρ) where r = response rate after stage 1 treatment m = avg number of SPs at each school ρ = inter-school correlation in outcome

D&I Adaptive Implementation Interventions Dec 2019 55 / 118

Sample Size Formulae for Cluster-Randomized SMARTs

For comparing 2 embedded AIs that begin with different treatments in a prototypical SMART

N ≥

4(z1−α/2+z1−β)

2

m∗δ2

× (2 − r) × (1 + (m − 1)ρ) where r = response rate after stage 1 treatment m = avg number of SPs at each school ρ = inter-school correlation in outcome

r = δ = 1/2, ρ = .03, m = 20, α = .05, need N = 100 for 80%power. Same question using standard 2-arm cluster trial requires N = 66.

D&I Adaptive Implementation Interventions Dec 2019 55 / 118

Recall the Autism SMART

D&I Adaptive Implementation Interventions Dec 2019 56 / 118

Recall the Results Comparing the 3 DTRs

Adaptive (a) TSCU (b) IJA Intervention AUC 95% CI AUC 95%CI (AAC,AAC+) 51.7 [43, 60] 9.5 [7.2,11.8] (JASP,AAC) 36.0 [28, 44] 7.2 [5.6,8.8] (JASP,JASP+) 33.1 [25, 42] 6.6 [5,8.2] No diff null p < 0.01 p < 0.05

D&I Adaptive Implementation Interventions Dec 2019 57 / 118

Recall the Example Marginal Mean Model for Longitudinal Outcomes

Yt : Total Socially Communicative Utterances at t = 0, 12, 24, 36. An example is the following piece-wise linear model for E[Yt(a1, a2)|X] : µi,(a1,a2)(Xi; β, η) = β0 + ηTX + 1t≤12{β1t + β2ta1} + 1t>12{12β1 + 12β2a1 + β3(t − 12) + β4(t − 12)a1 + β5(t − 12)a1a2} where X’s are mean-centered baseline covariates. Respects the fact that some embedded must AIs share trajectories up to the point of randomization. Other marginal mean models are possible, of course!

D&I Adaptive Implementation Interventions Dec 2019 58 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T Vi: working cov matrix for Yi under adaptive intervention (a1, a2). Ii,(a1,a2): indicator that person i has data consistent with adaptive intervention (a1, a2) (this is a function of (A1i, Ri, A2i)) Wi: diagonal matrix of the product of the inverse prob. of the

• bserved treatment (this is a function of (A1i, Ri, A2i)) =

(1/Pr(A1 | X)) × (1/Pr(A2 | X, A1, R)) (notation hack) next slide gives intuition for the weights

D&I Adaptive Implementation Interventions Dec 2019 59 / 118

Estimation

Intuition RE the weights W

D&I Adaptive Implementation Interventions Dec 2019 60 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T Vi: working cov matrix for Yi under adaptive intervention (a1, a2). Ii,(a1,a2): indicator that person i has data consistent with adaptive intervention (a1, a2) (this is a function of (A1i, Ri, A2i)) Wi: diagonal matrix of the product of the inverse prob. of the

• bserved treatment (this is a function of (A1i, Ri, A2i)) =

(1/Pr(A1 | X)) × (1/Pr(A2 | X, A1, R)) (notation hack) in the weights, why not use product of inverse-probs up to t

D&I Adaptive Implementation Interventions Dec 2019 61 / 118

For example, in the autism study, why not use ˜ Wi =     1 2 wi wi     instead of Wi =     wi wi wi wi     where wi = 2I{A1 = 1, R = 1} + 2I{A1 = −1} + 4I{A1 = 1, R = 0}?

D&I Adaptive Implementation Interventions Dec 2019 62 / 118

For example, in the autism study, why not use ˜ Wi =     1 2 wi wi     instead of Wi =     wi wi wi wi     ? in the weights, why use the product of the inverse probabilities? answer: with ˜ Wi weights and non-diagonal Vi, the cross-product terms in the estimating equations do not, in general, have mean zero

D&I Adaptive Implementation Interventions Dec 2019 63 / 118

For example, in the autism study, why not use ˜ Wi =     1 2 wi wi     instead of Wi =     wi wi wi wi     ? in the weights, why use the product of the inverse probabilities? answer: with ˜ Wi weights and non-diagonal Vi, the cross-product terms in the estimating equations do not, in general, have mean zero

Pepe, M. and Anderson G. (1994) A cautionary note on inference for marginal regression models with longitudinal data and general correlated response data. Communications in Statistics 23 (4), 939-951 Vansteelandt, S. (2007). On confounding, prediction and efficiency in the analysis of longitudinal and cross-sectional clustered data. Scandinavian Journal of Statistics 34 (3), 478-98. Tchetgen Tchetgen, E. J., M. M. Glymour, J. Weuve, and J. Robins (2012). Specifying the correlation structure in inverse-probability-weighting estimation for repeated measures. Epidemiology 23 (4), 644-46.

D&I Adaptive Implementation Interventions Dec 2019 63 / 118

For example, in the autism study, why not use ˜ Wi =     1 2 wi wi     instead of Wi =     wi wi wi wi     ? in the weights, why use the product of the inverse probabilities? answer: with ˜ Wi weights and non-diagonal Vi, the cross-product terms in the estimating equations do not, in general, have mean zero

Using Wi resolves this but not a good idea if Lots of decision points as in a micro-randomized trial; or Observational study settings where probabilities might be close to zero (multiplying the inverse of many near zero quantities will lead to very large weights) But these are not a concern in most SMARTs designed today.

D&I Adaptive Implementation Interventions Dec 2019 64 / 118

Some Open Problems I’d Like to Work on Next

That is, my next methods grant(s)

We are wrapping up Linear Mixed Models for comparing embedded DTRs Clustered SMART with a longitudinal outcome; “3 level analysis” Multi-level SMARTs

◮ Including designs that address spillover effects

R-conditional modeling and estimation to back out E[Yt(a1, a2)|X]

◮ Using a Structural Nested Mean Modeling approach

How best to elicit stakeholder conjectures about optimal DTR (tailoring variables, class of decision-rules): Graphical approaches? Clinical vignettes? Some smaller, but very interesting, lower hanging fruit: small sample adjustments to sandwich standard errors, estimating the weights for efficiency (guidance on choosing covariates for this), ...

D&I Adaptive Implementation Interventions Dec 2019 65 / 118

Shifting Gears a Bit: Why Mixed-Effect Models?

Indirect, yet intuitive, approach to posing working models for the marginal variance-covariance of Yi = ⇒ statistical efficiency Greater flexibility in choice of working var-cov models for designs with irregularly timed measurement occasions *** Secondary interest in (1) model-based predictions of the outcome trajectories (as opposed to noisier trajectories based on the individual’s observed repeated measures) and (2) some of the variance components

D&I Adaptive Implementation Interventions Dec 2019 66 / 118

A Mixed-Effect Model for the Embedded DTRs

Random Intercept Only

Y (a1,a2)

it

= µ(a1,a2)

it

+ e(a1,a2)

it

= βTX (a1,a2)

i,t

+ bi + ǫ(a1,a2)

it

where, for example, [Y (a1,a2)

i

| bi] ∼ N(µ(a1,a2)

i

+ (1, 1, 1, 1)Tbi, ν2

ǫ I4×4)

[bi] ∼ N(0, ν2

b)

which implies [Y (a1,a2)

i

] ∼ N(µ(a1,a2)

i

, ν2

b (1, 1, 1, 1)T(1, 1, 1, 1) + ν2 ǫ I4×4)

• r, if σ2 = ν2

b + ν2 ǫ , ρ = ν2 b/σ2, the marginal variance of Y (a1,a2) i

is V (a1,a2)

σ,ρ

= σ2     1 ρ ρ ρ ρ 1 ρ ρ ρ ρ 1 ρ ρ ρ ρ 1     = σ2Cρ

D&I Adaptive Implementation Interventions Dec 2019 67 / 118

A Mixed-Effect Model for the Embedded DTRs

Random Intercept Only

Y (a1,a2)

it

= µ(a1,a2)

it

+ e(a1,a2)

it

= βTX (a1,a2)

i,t

+ bi + ǫ(a1,a2)

it

where, for example, [Y (a1,a2)

i

| bi] ∼ N(µ(a1,a2)

i

+ (1, 1, 1, 1)Tbi, ν2

ǫ I4×4)

[bi] ∼ N(0, ν2

b)

[Y (a1,a2)

i

] ∼ N(µ(a1,a2)

i

, σ2Cρ) But we cannot maximize the marginal log-likelihood

n

• i

log ˜ fβ,σ2,ρ(Y (a1,a2)

i

) because we do not observe Y (a1,a2)

i

.

D&I Adaptive Implementation Interventions Dec 2019 68 / 118

A Mixed-Effect Model for the Embedded DTRs

Random Intercept Only

Y (a1,a2)

it

= µ(a1,a2)

it

+ e(a1,a2)

it

= βTX (a1,a2)

i,t

+ bi + ǫ(a1,a2)

it

where, for example, [Y (a1,a2)

i

| bi] ∼ N(µ(a1,a2)

i

+ (1, 1, 1, 1)Tbi, ν2

ǫ I4×4)

[bi] ∼ N(0, ν2

b)

[Y (a1,a2)

i

] ∼ N(µ(a1,a2)

i

, σ2Cρ) But we cannot maximize the marginal log-likelihood

n

• i

log ˜ fβ,σ2,ρ(Y (a1,a2)

i

) because we do not observe Y (a1,a2)

i

.

D&I Adaptive Implementation Interventions Dec 2019 69 / 118

A Mixed-Effect Model for the Embedded DTRs

Random Intercept Only

Y (a1,a2)

it

= µ(a1,a2)

it

+ e(a1,a2)

it

= βTX (a1,a2)

i,t

+ bi + ǫ(a1,a2)

it

So we propose to maximize a “pseudo log-likelihood” instead ˜ lβ,σ2,ρ(Yi) =

n

• i
• (a1,a2)

Wi Ii,(a1,a2) log ˜ fβ,σ2,ρ(Yi) where Yi is the observed longitudinal outcome, which leads to these estimating equations for β 0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)C −1

ρ Wi(Yi − µi,(a1,a2)(Xi; β, η)).

D&I Adaptive Implementation Interventions Dec 2019 70 / 118

A Mixed-Effect Model for the Embedded DTRs

Random Intercept Only

Y (a1,a2)

it

= µ(a1,a2)

it

+ e(a1,a2)

it

= βTX (a1,a2)

i,t

+ bi + ǫ(a1,a2)

it

[bi | Y (a1,a2)

i

] ∼ Normal with posterior mean = ρ (1, 1, 1, 1)C −1

ρ

• Y (a1,a2)

i

− βTX (a1,a2)

i,t

• But, again, we do not have Y (a1,a2)

i

for each person! So we propose ˆ bi = arg max

bi

• a1,a2

WiIi,(a1,a2) log ˜ f (bi | Yi) =

• a1,a2 WiIi,(a1,a2)
• ρ (1, 1, 1, 1)C −1

ρ

• Yi − βTX (a1,a2)

i,t

• a1,a2 WiIi,(a1,a2)

D&I Adaptive Implementation Interventions Dec 2019 71 / 118

Extra, Back-pocket Slides; Some More Technical

D&I Adaptive Implementation Interventions Dec 2019 73 / 118

Estimation

But, first, let’s review the observed data...

D&I Adaptive Implementation Interventions Dec 2019 74 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T

D&I Adaptive Implementation Interventions Dec 2019 75 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi;

D&I Adaptive Implementation Interventions Dec 2019 75 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T

D&I Adaptive Implementation Interventions Dec 2019 75 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T Vi: working cov matrix for Yi under adaptive intervention (a1, a2).

D&I Adaptive Implementation Interventions Dec 2019 75 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T Vi: working cov matrix for Yi under adaptive intervention (a1, a2). Ii,(a1,a2): indicator that person i has data consistent with adaptive intervention (a1, a2) (this is a function of (A1i, Ri, A2i))

D&I Adaptive Implementation Interventions Dec 2019 75 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T Vi: working cov matrix for Yi under adaptive intervention (a1, a2). Ii,(a1,a2): indicator that person i has data consistent with adaptive intervention (a1, a2) (this is a function of (A1i, Ri, A2i)) Wi: diagonal matrix of the product of the inverse prob. of the

• bserved treatment in stages 1 and 2 (this is a function of

(A1i, Ri, A2i)) next slide gives intuition for the weights

D&I Adaptive Implementation Interventions Dec 2019 75 / 118

Estimation

Intuition RE the weights W

D&I Adaptive Implementation Interventions Dec 2019 76 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T Vi: working cov matrix for Yi under adaptive intervention (a1, a2). Ii,(a1,a2): indicator that person i has data consistent with adaptive intervention (a1, a2) (this is a function of (A1i, Ri, A2i)) Wi: diagonal matrix of the product of the inverse prob. of the

• bserved treatment in stages 1 and 2 (this is a function of

(A1i, Ri, A2i)) in the weights, why use the product of the inverse probabilities?

D&I Adaptive Implementation Interventions Dec 2019 77 / 118

For example, in the autism study, why not use ˜ Wi =     1 2 wi wi     instead of Wi =     wi wi wi wi     where wi = 2I{A1 = 1, R = 1} + 2I{A1 = −1} + 4I{A1 = 1, R = 0}?

D&I Adaptive Implementation Interventions Dec 2019 78 / 118

An Estimating Equation

0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

Yi: observed longitudinal outcomes, e.g., (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean traj. under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T Vi: working cov matrix for Yi under adaptive intervention (a1, a2). Ii,(a1,a2): indicator that person i is consistent with (a1, a2) Wi: weight matrix, a function of the inverse prob. of the observed treatment in stages 1 and 2 in the weights, why use the product of the inverse probabilities? answer: because of the non-diagonal Vi

D&I Adaptive Implementation Interventions Dec 2019 79 / 118

You may be wondering about sample size/power?

Special Issue in Journal of Clinical Child and Adolescent Psychology APA’s Division 53 Journal

Adaptive Interventions in Child and Adolescent Mental Health Editors: Daniel Almirall and Andrea Chronis-Tuscano Topics: Over 10 blinded, externally peer-reviewed papers covering anxiety, depression, autism, prevention, ADHD, child obesity Discussion: Dr. Joel Sherrill, NIMH Division of Services and Interventions Research, NIMH

D&I Adaptive Implementation Interventions Dec 2019 81 / 118

SMART Case Study #4: Adaptive Implementation of Effective Programs (ADEPT)

D&I Adaptive Implementation Interventions Dec 2019 82 / 118

Adaptive Implementation Intervention in Mental Health

PI: Kilbourne; Co-I: Almirall (CO/AR/MI; Aim is to improve uptake of psychosocial intervention for mood disorders; primary aim compared initial REP+EF vs REP+EF+IF.)

Non-responding site if: < 50% of previously identified patients were

• ffered at least three LG sessions (≥ 3 out of 6)

Estimation

Intuition RE the I and W

D&I Adaptive Implementation Interventions Dec 2019 84 / 118

Definition of an Adaptive Intervention, in symbols

{S1, a1, S2(a1), a2, . . . , ST(¯ aT−1), aT} St is the state or status of the individual/unit at time t and at indexes a possible action (treatment) at time t

◮ e.g., intensify medication dose? ◮ e.g., add medication to behavioral intervention? ◮ e.g., continue treatment and monitor?

An adaptive intervention is a sequence of decision rules {d1(s1), d2(s1, a1, s2), . . . , dT(¯ aT−1, ¯ sT)}.

D&I Adaptive Implementation Interventions Dec 2019 85 / 118

Interventions for Minimally Verbal Children with Autism

PIs: Kasari(UCLA), Almirall(Mich), Kaiser(Vanderbilt), Smith(Rochester), Lord(Cornell)

Primary and Secondary Aims

Primary Aim: What is the best first-stage treatment in terms of spoken communication at week 24: JASP vs DTT? (Sized N = 192 for this aim; compares A+B+C+D vs E+F+G+H) Secondary Aim 1: Determine whether adding a parent training provides additional benefit among children who demonstrate a positive early response to either JASP or DTT (D+H vs C+G). Secondary Aim 2: Determine whether adding JASP+DTT provides additional benefit among children who demonstrate a slow early response to either JASP or DTT (A+E vs B+F). Secondary Aim 3: Compare eight pre-specified adaptive interventions. [Note, we can now compare always JASP vs always DTT!]

D&I Adaptive Implementation Interventions Dec 2019 87 / 118

Challenges in the Conduct of this Initial Autism SMART

Slow responder rate, while based on prior data, was lower than anticipated during the design of the trial. Responder/Slow-responder measure could be improved to make more useful in actual practice. There was some disconnect with the definition of slow-response status and the therapist’s clinical judgment.

D&I Adaptive Implementation Interventions Dec 2019 88 / 118

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2

D&I Adaptive Implementation Interventions Dec 2019 89 / 118

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 E[Y (1, 1)] = β0 + β1 + β2 = (JASP,JASP+) E[Y (1, −1)] = β0 + β1 − β2 = (JASP,AAC) E[Y (−1, .)] = β0 − β1 = (AAC,AAC+)

D&I Adaptive Implementation Interventions Dec 2019 89 / 118

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 −2β1 + β2 = (AAC,AAC+) vs (JASP,JASP+) −2β1 − β2 = (AAC,AAC+) vs (JASP,AAC) −2β2 = (JASP,AAC) vs (JASP,JASP+)

D&I Adaptive Implementation Interventions Dec 2019 90 / 118

How Do We Estimate this Marginal Model?

E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 The observed data is {Xi, A1i, Ri, A2i, Yi}, i = 1, . . . , N. Regressing Y on [1, X, A1, I(A1 = 1)A2] often won’t work. Why? By design, there is an imbalance in the types individuals following AI#1 vs AI#3 (for example)? This imbalance is due to a post-randomization variable R. Adding R to this regression does not fix this and may make it worse! How do we account for the fact that responders to JASP are consistent with two of the embedded AIs? We use something called weighted-and-replicated regression. It is easy!

D&I Adaptive Implementation Interventions Dec 2019 91 / 118

Before Weighting-and-Replicating

D&I Adaptive Implementation Interventions Dec 2019 92 / 118

After Weighting-and-Replicating

D&I Adaptive Implementation Interventions Dec 2019 93 / 118

Weighted-and-Replicated Regression Estimator (WRR)

Statistical foundation found in work by Orellana, Rotnitzky and Robins: Robins JM, Orellana L, Rotnitzky A. Estimation and extrapolation in

• ptimal treatment and testing strategies. Statistics in Medicine. 2008

Jul; 27:4678-4721. Orellana L, Rotnitzky A, Robins JM. Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part I: Main Content. Int J Biostat. 2010; 6(2): Article No. 8. (...ditto...), Part II: Proofs of Results. Int J Biostat. 2010;6(2): Article No. 9. 4678-4721. Very nicely explained and implemented with SMART data in: Nahum-Shani I, Qian M, Almirall D, et al. Experimental design and primary data analysis methods for comparing adaptive interventions. Psychol Methods. 2012 Dec; 17(4): 457-77.

D&I Adaptive Implementation Interventions Dec 2019 94 / 118

Weighted-and-Replicated Regression Estimator (WRR)

Weighting (IPTW): By design, each individual/unit has a different probability of following the sequence of treatment s/he was offered (weights account for this)

◮ e.g., W = 2I{A1 = 1, R = 1} + 2I{A1 = −1} + 4I{A1 = 1, R = 0}.

Replication: Some individuals may be consistent with multiple embedded regimes (replication takes advantage of this and permits pooling covariate information)

◮ e.g., Replicate (double) the responders to JASP: assign A2 = 1 to half

and A2 = −1 to the other half

◮ e.g., The new data set is of size M = N + I{A1 = 1, R = 1}

Implementation is as easy as running a weighted least squares: (ˆ η, ˆ β) = arg min

η,β

1 M

M

• i=1

Wi(Yi − µ(Xi, A1i, A2i; η, β))2. SE’s: Use ASEs to account for weighting/replicating (or bootstrap).

D&I Adaptive Implementation Interventions Dec 2019 95 / 118

An Interesting Connection Between Estimators

Recall Robins’ G-Computation Estimator (not to be confused with the G-Estimator which is an entirely different thing!:)

• E[Y (1, 1)] =

E[Y |A] Pr[R = 1|JASP] + E[Y |C](1 − Pr[R = 1|JASP])

• E[Y (1, −1)] =

E[Y |A] Pr[R = 1|JASP] + E[Y |B](1 − Pr[R = 1|JASP])

• E[Y (−1, .)] =

E[Y |D] Pr[R = 1|AAC] + E[Y |E](1 − Pr[R = 1|AAC]) This estimator is algebraically identical to fitting the WRR Estimator with no covariates and sample-proportion estimated weights (rather than the known true weights). Comparing these two provides a way to compare the added-value of adjusting for covariates in terms of statistical efficiency gains.

D&I Adaptive Implementation Interventions Dec 2019 96 / 118

Results from an Analysis of the Autism SMART

Recall: N = 61, and the primary outcome is SCU at Week 24 (SD=34.6).

WRR with no Covts WRR with Covts and with SAMPLE and Known Wt PROP Wt (G-Comp) ESTIMAND EST SE PVAL EST SE PVAL (AAC,AAC+) 60.5 5.8 < 0.01 61.0 6.0 < 0.01 (JASP,AAC) 42.6 4.9 < 0.01 38.2 6.9 < 0.01 (JASP,JASP+) 36.3 5.0 < 0.01 40.0 8.0 < 0.01 (AAC,AAC+) vs (JASP,JASP+) 24.3 7.9 < 0.01 21.0 10.2 0.04 (AAC,AAC+) vs (JASP,AAC) 17.9 8.2 0.03 22.8 9.4 0.02 (JASP,AAC) vs (JASP,JASP+) 6.4 3.8 0.10

• 1.8

7.7 0.82 What’s the lesson? The regression approach is more useful. (And, it is a good idea to adjust for baseline covariates!) Of course, this is well-known. But the story gets even more interesting...

D&I Adaptive Implementation Interventions Dec 2019 97 / 118

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }.

D&I Adaptive Implementation Interventions Dec 2019 98 / 118

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }. However, from work by Robins and colleagues (1995; also, Hirano et al (2003)), there are gains in statistical efficiency if using an WRR with weights that are estimated using auxiliary baseline (L1) and time-varying (L2) covariate information. Here’s how to do it with the autism SMART: The observed data is now {L1i, Xi, A1i, Ri, L2i, A2i, Yi} Use logistic regression to get p1 = Pr(A1 | L1, X) Use logistic regression to get p2 = Pr(A2 | L1, X, A1 = 1, R = 0, L2). Use W = I{A1 = 1, R = 0}/( p1 p2) + I{ everyone else }/ p1.

D&I Adaptive Implementation Interventions Dec 2019 98 / 118

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }. However, from work by Robins and colleagues (1995; also, Hirano et al (2003)), there are gains in statistical efficiency if using an WRR with weights that are estimated using auxiliary baseline (L1) and time-varying (L2) covariate information. Here’s how to do it with the autism SMART: The observed data is now {L1i, Xi, A1i, Ri, L2i, A2i, Yi} Use logistic regression to get p1 = Pr(A1 | L1, X) Use logistic regression to get p2 = Pr(A2 | L1, X, A1 = 1, R = 0, L2). Use W = I{A1 = 1, R = 0}/( p1 p2) + I{ everyone else }/ p1. The key is to choose Lt’s that are highly correlated with Y !

D&I Adaptive Implementation Interventions Dec 2019 98 / 118

Sim: Relative RMSE for (AAC,AAC+) vs (JASP,JASP+)

D&I Adaptive Implementation Interventions Dec 2019 99 / 118

Results from an Analysis of the Autism SMART

Recall: N = 61, and the primary outcome is SCU at Week 24 (SD=34.6).

WRR with Covts WRR with Covts and Known Wt and Covt-Est Wt ESTIMAND EST SE PVAL EST SE PVAL (AAC,AAC+) 60.5 5.8 < 0.01 60.2 5.6 < 0.01 (JASP,AAC) 42.6 4.9 < 0.01 43.1 4.5 < 0.01 (JASP,JASP+) 36.3 5.0 < 0.01 35.4 4.4 < 0.01 (AAC,AAC+) vs (JASP,JASP+) 24.3 7.9 < 0.01 24.9 7.4 < 0.01 (AAC,AAC+) vs (JASP,AAC) 17.9 8.2 0.03 17.1 7.9 0.03 (JASP,AAC) vs (JASP,JASP+) 6.4 3.8 0.10 7.7 3.0 0.01 The WRR implementation with covariates and covariate-estimated weights permits us to obtain scientific information from a SMART with less uncertainty.

D&I Adaptive Implementation Interventions Dec 2019 100 / 118

Rule-of-thumb concerning which auxiliary variables to use in the WRR for comparing embedded of AIs in a SMART.

Key is to include in Lt variables which are (highly) correlated with Y , even if not of scientific interest. A potentially useful rule-of-thumb (not dogma): Include in L1, all variables that were used to stratify the initial randomization. Include in L2, all variables that were used to stratify the second randomization. Let the science dictate which X’s to include in the final regression model.

◮ e.g., Investigator may be interested in whether baseline levels of spoken

communication moderate the effect of JASP vs JASP+AAC.

◮ Of course: It is possible for X = L1, but not possible for X to include

any post-A1 measures.

D&I Adaptive Implementation Interventions Dec 2019 101 / 118

Challenges to Address in Longitudinal Setting

Modeling Considerations: The intermixing of repeated measures and sequential randomizations requires new modeling considerations to account for the fact that embedded AIs will share paths/trajectories at different time points (this is non-trivial) Implications for Interpreting Longitudinal Models: (1) Comparison of slopes is no longer appropriate in many circumstances; (2) Need for new, clinically relevant, easy-to-understand summary measures of the mean trajectories over time Statistical: Develop an estimator that takes advantage of the within person correlation in the outcome over time

D&I Adaptive Implementation Interventions Dec 2019 102 / 118

An Example Marginal Model for Longitudinal Outcomes

Yt : # Socially Communicative Utterances at week t. t = 0, 12, 24, 36 The comparison of embedded AIs with longitudinal data arising from a SMART will require longitudinal models that permit deflections in trajectories and respect the fact that some embedded AIs will share paths/trajectories up to the point of randomization. An example is the following piece-wise linear model: E[Yt(a1, a2)|X] = β0 + ηTX + 1t≤12{β1t + β2ta1} + 1t>12{12β1 + 12β2a1 + β3(t − 12) + β4(t − 12)a1 + β5(t − 12)a1a2} where X’s are mean-centered baseline covariates.

D&I Adaptive Implementation Interventions Dec 2019 103 / 118

Modeling Considerations

Regime (-1,0): (AAC, AAC+) 12 24 36 t Y

• β0
• slope =

β1 − β2 slope = β3 − β4

D&I Adaptive Implementation Interventions Dec 2019 104 / 118

Modeling Considerations

Regime (1,1): (JASP, JASP+) 12 24 36 t Y

• slope =

β1 + β2 slope = β3 + β4 + β5

D&I Adaptive Implementation Interventions Dec 2019 105 / 118

Modeling Considerations

Regime (1,-1): (JASP, AAC) 12 24 36 t Y

• slope =

β1 + β2 slope = β3 + β4 − β5

D&I Adaptive Implementation Interventions Dec 2019 106 / 118

Implications of New Modeling Considerations for Summarizing each AI

Potential Solution: Summarize each AI by the area under the curve (during an interval chosen by the investigator) Clinical advantage: AUC is easy to understand clinically; it is the average of the primary outcome over a specific interval of time Statistical inference is easy: AUC is linear function of parameters (β’s) in marginal model

D&I Adaptive Implementation Interventions Dec 2019 107 / 118

Statistical: WRR Estimator for Longitudinal Outcomes

We use the following estimating equation to estimate marginal model for longitudinal outcomes: 0 = 1 n

n

• i=1
• (a1,a2)

Ii,(a1,a2) ˙ µ(Xi)Vi

−1 (a1,a2)Wi(Yi − µi,(a1,a2)(Xi; β, η)),

where Yi: observed longitudinal outcomes, i.e. (Yi,0, Yi,12, Yi,24, Yi,36)T µi mean trajectory under adaptive intervention (a1, a2) conditnl on Xi; ˙ µ(Xi): the design matrix, i.e.

• ∂µi (Xi,(a1,a2);β,η)

∂(β,η)T

T Wi: a diagonal matrix containing inverse probability of following the

• ffered treatment sequence at each time point (function of Ri);

Vi: working cov matrix for Yi under adaptive intervention (a1, a2). Ii,(a1,a2): indicator that person i has data consistent with adaptive intervention (a1, a2) (function of Ri)

D&I Adaptive Implementation Interventions Dec 2019 108 / 118

Child Attention Deficit Hyperactivity Disorder (ADHD)

PI: Pelham (FIU) (N = 153; ages 6-12; 8 month study; monthly non-response based on two teacher ratings (ITB < 0.75 and IRS > 1 domain)

D&I Adaptive Implementation Interventions Dec 2019 109 / 118

Analysis of Longitudinal Outcomes in the ADHD SMART

Average classroom performance

• ver the school year for each AI

AI Estimate SE (BMD,BMD+) 21.4 0.91 (BMD,BMD+MED) 21.3 0.95 (MED, MED+BMD) 19.2 0.96 (MED, MED+) 19.0 0.85

D&I Adaptive Implementation Interventions Dec 2019 110 / 118

Adaptive Implementation Intervention in Mental Health

PI: Kilbourne; Co-I: Almirall (CO/AR/MI; Aim is to improve uptake of psychosocial intervention for mood disorders; primary aim compared initial REP+EF vs REP+EF+IF.)

Example of an Adaptive Intervention in Autism

Some Background First... ≥50% of children with autism who received interventions beginning at age 2 remained non-verbal at age 9 Failure to develop spoken language by age 5 = poor prognosis

D&I Adaptive Implementation Interventions Dec 2019 112 / 118

Example of an Adaptive Intervention in Autism

Some Background First... ≥50% of children with autism who received interventions beginning at age 2 remained non-verbal at age 9 Failure to develop spoken language by age 5 = poor prognosis Evidence Base:

◮ Discrete Trials Training, ◮ Joint Attention, Symbolic Play, Engagement & Regulation (JASPER) ◮ Enhanced Milieu Teaching (EMT)

Promising: Augmentative, Alternative Communication (AAC) devices But AAC’s are costly & not all children need it.

D&I Adaptive Implementation Interventions Dec 2019 112 / 118

Example of an Adaptive Intervention in Autism

Some Background First... ≥50% of children with autism who received interventions beginning at age 2 remained non-verbal at age 9 Failure to develop spoken language by age 5 = poor prognosis Evidence Base:

◮ Discrete Trials Training, ◮ Joint Attention, Symbolic Play, Engagement & Regulation (JASPER) ◮ Enhanced Milieu Teaching (EMT)

Promising: Augmentative, Alternative Communication (AAC) devices But AAC’s are costly & not all children need it.

◮ Research is limited. Mostly single-subject studies. No rigorous trials.

Motivation for an adaptive intervention involving AAC’s in context

• f JASPER-EMT among older, minimally-verbal children with

autism.

D&I Adaptive Implementation Interventions Dec 2019 112 / 118

Example of an Adaptive Intervention in Autism

For minimally verbal children with autism spectrum disorder Stage One JASP+EMT for 12 weeks; Stage Two At the end of week 12, determine early sign of response:

◮ IF slow responder: Augment JASP+EMT with AAC for 12 weeks; ◮ ELSE IF responder: Maintain JASP+EMT for 12 weeks. D&I Adaptive Implementation Interventions Dec 2019 113 / 118

Example of an Adaptive Intervention in Autism

For minimally verbal children with autism spectrum disorder Stage One JASP+EMT for 12 weeks; Stage Two At the end of week 12, determine early sign of response:

◮ IF slow responder: Augment JASP+EMT with AAC for 12 weeks; ◮ ELSE IF responder: Maintain JASP+EMT for 12 weeks. D&I Adaptive Implementation Interventions Dec 2019 113 / 118

How was response/slow-response defined?

Percent change from baseline to week 12 was calculated for 7 variables: socially communicative utterances (SCU), SCU/total comments, mean length utterance, total word roots, words per minute, total comments, unique word combinations Fast Responder: if ≥25% change on 7 measures; Slower Responder: otherwise (this includes kids with no improvement, which is rare)

D&I Adaptive Implementation Interventions Dec 2019 114 / 118

SMART Case Study: Characterizing Cognition in Non-verbal Individuals with ASD (CCNIA)

D&I Adaptive Implementation Interventions Dec 2019 115 / 118

Example of a (first-ever) SMART in Autism Research

PI: Kasari (UCLA)

The population of interest: Children with autism spectrum disorder Age: 5-8 Minimally verbal: <20 spontaneous words in a 20-min. language test History of treatment: ≥2 years of prior intervention Functioning: ≥2 year-old on non-verbal intelligence tests

D&I Adaptive Implementation Interventions Dec 2019 116 / 118

Example of a SMART in Autism Research (N = 61)

PI: Kasari (UCLA)

D&I Adaptive Implementation Interventions Dec 2019 117 / 118

Three AIs “Embedded” in this Example Autism SMART

D&I Adaptive Implementation Interventions Dec 2019 118 / 118

Three AIs “Embedded” in this Example Autism SMART

(JASP,JASP+) (JASP,AAC) (AAC,AAC+)

D&I Adaptive Implementation Interventions Dec 2019 118 / 118