A Look Under the Hood of SMART Designs for Developing Adaptive - - PowerPoint PPT Presentation

A Look Under the Hood of SMART Designs for Developing Adaptive Interventions

Daniel Almirall, PhD Survey Research Center, Institute for Social Research University of Michigan April 11, 2016 Center for Drug Use and HIV Research New York, NY

Almirall and many friends Adaptive Interventions April 2016 1 / 66

I don’t do any of this research by myself.

Students Xi Lu, Penn State Colleagues Billie Nahum-Shani, Univ Mich Susan A. Murphy, Univ Mich Connie Kasari, UCLA (autism) Amy Kilbourne, Univ Mich (implementation science) Kevin Lynch, Univ Penn And many others...

Almirall and many friends Adaptive Interventions April 2016 2 / 66

Outline

Adaptive Interventions Sequential Multiple Assignment Randomized Trials (SMART) Longitudinal Analysis of a (first) SMART in Autism 5 SMART Case Studies (all in the field or in data analysis mode)

◮ Adaptive Interventions for Minimally Verbal Children (AIM-ASD) ◮ Adaptive Implementation of Effective Programs (ADEPT, mood dx) ◮ Extending Treatment Effectiveness in Adult Alcoholism (ExTEnD) ◮ Treatment for Pregnant Women with Heroine Dependence (RBT) ◮ Getting SMART about Social & Academic Engagement (ASD Schools)

Myths or Misconceptions about Adaptive Interventions and SMARTs

Almirall and many friends Adaptive Interventions April 2016 3 / 66

Sequential, Individualized Treatment Often Needed in Mental Health

Intervention often entails a sequential, individualized approach whereby treatment is adapted and re-adapted over time in response to the specific needs and evolving status of the individual. This type of sequential decision-making is necessary when there is high level of individual heterogeneity in response to treatment.

◮ e.g., what works for one individual may not work for another ◮ e.g., what works now may not work later ◮ e.g., for some, what appears not to work in the short-run has positive

long-term consequences

Adaptive interventions help guide this type of individualized, sequential, treatment decision making

Almirall and many friends Adaptive Interventions April 2016 4 / 66

Adaptive Interventions

Almirall and many friends Adaptive Interventions April 2016 5 / 66

Definition of an Adaptive Intervention

Adaptive Interventions (AI) provide one way to operationalize the strategies (e.g., continue, augment, switch, step-down) leading to individualized sequences of treatment. A sequence of decision rules that specify whether, how, when (timing), and based on which measures, to alter the dosage (duration, frequency or amount), type, or delivery of treatment(s) at decision stages in the course of care.

Almirall and many friends Adaptive Interventions April 2016 6 / 66

Example of an Adaptive Intervention in Alcoholism

Stage One NTX + MM + Monitor weekly for 2+ HDD/week; Stage Two IF patient trigger’s a non-response in weeks 2-8

◮ THEN Augment with Cognitive Behavioral Intervention (CBI); ◮ ELSE IF continued responder until week 8 ⋆ THEN NTX + Phone Counseling to maintain response; Almirall and many friends Adaptive Interventions April 2016 7 / 66

Example of an Adaptive Intervention in Alcoholism

Stage One NTX + MM + Monitor weekly for 2+ HDD/week; Stage Two IF patient trigger’s a non-response in weeks 2-8

◮ THEN Augment with Cognitive Behavioral Intervention (CBI); ◮ ELSE IF continued responder until week 8 ⋆ THEN NTX + Phone Counseling to maintain response; Almirall and many friends Adaptive Interventions April 2016 7 / 66

Many Unanswered Questions when Building a High-Quality Adaptive Intervention.

Why 2+ HDDs per week? Why not, instead, 5+ HDDs per week? Why should responders transition at week 8 to maintenance treatment? For continued responders at week 8, what is the effect of providing Phone Counseling? Do we really need it? Insufficient empirical evidence or theory to address such questions. In the past: relied on expert opinion, clinical expertise, or piecing together an AI with separate RCTs (e.g., practice parameters) Sequential Multiple Assignment Randomized Trials (SMARTs) address such questions empirically, using experimental design principles.

Almirall and many friends Adaptive Interventions April 2016 8 / 66

Example of an Adaptive Intervention in Autism

For minimally verbal children with autism spectrum disorder Stage One JASP for 12 weeks; Stage Two At the end of week 12, determine early sign of response:

◮ IF slow responder: Augment JASP with AAC for 12 weeks; ◮ ELSE IF responder: Maintain JASP for 12 weeks.

‐ ‐ ‐ ‐

Almirall and many friends Adaptive Interventions April 2016 9 / 66

Example of an Adaptive Intervention in Autism

For minimally verbal children with autism spectrum disorder Stage One JASP for 12 weeks; Stage Two At the end of week 12, determine early sign of response:

◮ IF slow responder: Augment JASP with AAC for 12 weeks; ◮ ELSE IF responder: Maintain JASP for 12 weeks.

Continue: JASP Responders JASP Augment: JASP + AAC Slow Responders

First‐stage Treatment (Weeks 1‐12) Second‐stage Treatment (Weeks 13‐24) End of Week 12 Responder Status

Almirall and many friends Adaptive Interventions April 2016 9 / 66

How was response/slow-response defined?

Percent change from baseline to week 12 was calculated for 7 variables: 7 variables: socially communicative utterances (SCU), percent SCU, mean length utterance, total word roots, words per minute, total comments, unique word combinations Fast Responder: if ≥25% change on 7 measures; Slower Responder: otherwise (includes kids with no improvement)

Almirall and many friends Adaptive Interventions April 2016 10 / 66

Many Unanswered Questions when Building a High-Quality Adaptive Intervention.

Is it better to provide AAC from the start? Who benefits from initial AAC versus who benefits from delayed AAC? For slow responders, what is the effect of providing the AAC vs intensifying JASP (not providing AAC)? Insufficient empirical evidence or theory to address such questions. In the past: relied on expert opinion, clinical expertise, or piecing together an AI with separate RCTs (e.g., practice parameters) Sequential Multiple Assignment Randomized Trials (SMARTs) address such questions empirically, using experimental design principles.

Almirall and many friends Adaptive Interventions April 2016 11 / 66

Sequential Multiple Assignment Randomized Trials (SMART)

Almirall and many friends Adaptive Interventions April 2016 12 / 66

What is a Sequential Multiple Assignment Randomized Trial (SMART)?

A type of multi-stage randomized trial design. At each stage, subjects randomized to a set of feasible/ethical treatment options. Treatment options at latter stages may be restricted by response to earlier treatments.

Almirall and many friends Adaptive Interventions April 2016 13 / 66

What is a Sequential Multiple Assignment Randomized Trial (SMART)?

A type of multi-stage randomized trial design. At each stage, subjects randomized to a set of feasible/ethical treatment options. Treatment options at latter stages may be restricted by response to earlier treatments. SMARTs were developed explicitly for the purpose of building a high-quality Adaptive Intervention.

Almirall and many friends Adaptive Interventions April 2016 13 / 66

Example of a SMART in Autism Research

The population of interest: Children with autism spectrum disorder Age: 5-8 Minimally verbal: <20 spontaneous words in a 20-min. language test History of treatment: ≥2 years of prior intervention Functioning: ≥2 year-old on non-verbal intelligence tests

Almirall and many friends Adaptive Interventions April 2016 14 / 66

Example of a SMART in Autism Research (N = 61)

PI: Kasari (UCLA).

Almirall and many friends Adaptive Interventions April 2016 15 / 66

There are 3 AIs Embedded in this Example Autism SMART

(JASP,JASP+) Subgroups A+C (JASP,AAC) Subgroups A+B (AAC,AAC+) Subgroups D+E

Almirall and many friends Adaptive Interventions April 2016 16 / 66

SMARTs permit scientists to answer many interesting questions for building a high-quality adaptive intervention.

Primary Aim: What is the best first-stage treatment in terms of spoken communication at week 24: JASP alone vs JASP+AAC?

Almirall and many friends Adaptive Interventions April 2016 17 / 66

SMARTs permit scientists to answer many interesting questions for building a high-quality adaptive intervention.

Primary Aim: What is the best first-stage treatment in terms of spoken communication at week 24: JASP alone vs JASP+AAC? Secondary Aim: Which is the best of the three adaptive interventions embedded in this SMART? I will show you results for the Secondary Aim: Compare 3 adaptive interventions based on a longitudinal outcome.

Almirall and many friends Adaptive Interventions April 2016 17 / 66

There are 3 AIs Embedded in this Example Autism SMART

(JASP,JASP+) Subgroups A+C (JASP,AAC) Subgroups A+B (AAC,AAC+) Subgroups D+E

Almirall and many friends Adaptive Interventions April 2016 18 / 66

Longitudinal Analyses

Almirall and many friends Adaptive Interventions April 2016 19 / 66

Longitudinal Outcomes in the Autism SMART

Outcomes collected at baseline, and weeks 12, 24 and 36 Primary outcome (verbal): from 20-minute “natural language sample”: Total spontaneous communicative utterances (TSCU) Secondary outcome (non-linguistic): Initiating joint attention (IJA; e.g., pointing; JASP mechanism)

Almirall and many friends Adaptive Interventions April 2016 20 / 66

Results

Adaptive (a) TSCU (b) IJA Intervention AUC 95% CI AUC 95%CI (AAC,AAC+) 51.7 [43, 60] 9.5 [7.2,11.8] (JASP,AAC) 36.0 [28, 44] 7.2 [5.6,8.8] (JASP,JASP+) 33.1 [25, 42] 6.6 [5,8.2] No diff null p < 0.01 p < 0.05

Almirall and many friends Adaptive Interventions April 2016 21 / 66

Looking Under the Hood of Various SMART Case Studies

Almirall and many friends Adaptive Interventions April 2016 22 / 66

SMART Case Study #1: Adaptive Interventions for Minimally Verbal Children with ASD (AIM-ASD)

Almirall and many friends Adaptive Interventions April 2016 23 / 66

Interventions for Minimally Verbal Children with Autism

PIs: Kasari(UCLA), Almirall(Mich), Kaiser(Vanderbilt), Smith(Rochester), Lord(Cornell)

What the original study did not aim to examine?

But in post-funding conversations, there was great interest in the effect of JASP+DTT!

Interventions for Minimally Verbal Children with Autism

PIs: Kasari(UCLA), Almirall(Mich), Kaiser(Vanderbilt), Smith(Rochester), Lord(Cornell)

! ! ! ! ! !

Subgroup!

A! B! C! D! E! Non0Responders!

(Parent!training!no! feasible)!

JASP!(joint! attention!and! social!play)! Continue!JASP! JASP!+!Parent! Training!

R!

DTT!(discrete! trials!training)! Continue!DTT! DTT!+!Parent! Training! Responders!

(Blended!txt! unnecessary)!

R!

Non0Responders!

(Parent!training!not! feasible)!

Responders!

(Blended!txt! unnecessary)!

R!

JASP!+!DTT! Continue!JASP!

R!

JASP!+!DTT! Continue!DTT!

R!

F! G! H!

Primary and Secondary Aims

Primary Aim: What is the best first-stage treatment in terms of spoken communication at week 24: JASP vs DTT? (Sized N = 192 for this aim; compares A+B+C+D vs E+F+G+H) Secondary Aim 1: Determine whether adding a parent training provides additional benefit among children who demonstrate a positive early response to either JASP or DTT (D+H vs C+G). Secondary Aim 2: Determine whether adding JASP+DTT provides additional benefit among children who demonstrate a slow early response to either JASP or DTT (A+E vs B+F). Secondary Aim 3: Compare eight pre-specified adaptive interventions. [Note, we can now compare always JASP vs always DTT!]

Almirall and many friends Adaptive Interventions April 2016 27 / 66

SMART Case Study #2: Adaptive Implementation of Effective Programs (ADEPT) in Mood Disorders

Almirall and many friends Adaptive Interventions April 2016 28 / 66

Adaptive Implementation Intervention in Mental Health

PI: Kilbourne; Co-I: Almirall (CO/AR/MI; Aim is to improve uptake of psychosocial intervention for mood disorders; primary aim compared initial REP+EF vs REP+EF+IF.)

SMART Case Study #3: ExTEnd Study in Adult Alcoholism

Almirall and many friends Adaptive Interventions April 2016 30 / 66

Extending Treatment Effectiveness in Alcohol Dependence

PIs: David Oslin (N = 302)

SMART Case Study #4: Treatment for Pregnant Women with Heroine Dependence

Almirall and many friends Adaptive Interventions April 2016 32 / 66

Extending Treatment Effectiveness in Alcohol Dependence

PIs: Hendree Jones (N = 300)

SMART Case Study #5: Getting SMART about Social and Academic Engagement in Elementary Aged Children with ASD

Almirall and many friends Adaptive Interventions April 2016 34 / 66

Academic & Social Engagement in School-Children w/ASD

PIs: Kasari; Co-I: Almirall; IES-funded Pilot SMART

Primary and Secondary Aims of this Pilot SMART

Primary Aim: Address feasibility and acceptability concerns related to the embedded adaptive interventions

◮ identifying children as early vs. slower responders by the

paraprofessionals in the context of RR,

◮ transitioning children to Parent or Peer at wk12, ◮ providing augmented Peer+Parent to slower responders ◮ not providing augmented treatment to early responders at wk20 ◮ satisfaction with txt sequences by children, parents, teachers,

paraprofessionals & school champions

◮ teacher-rated measures of progress during CS for deciding Parent vs

Peer

Secondary Aim: To obtain preliminary data to support a fully-powered SMART.

Almirall and many friends Adaptive Interventions April 2016 36 / 66

Myths and Misconceptions about Adaptive Interventions

Tailoring variables cannot differ based on previous intervention An adaptive intervention must recommend a single intervention component at each decision point Adaptive interventions seek to replace clinical judgement Adaptive interventions are only relevant in treatment settings Adaptive interventions are non-standard because they involve randomization

Almirall and many friends Adaptive Interventions April 2016 37 / 66

Myths and Misconceptions about SMART Studies

SMARTs require prohibitively large sample sizes All SMARTs require Multiple-Comparisons Adjustments All research on adaptive interventions requires a SMART All SMARTs must include an embedded tailoring variable All aspects of an embedded adaptive intervention must be randomized SMARTs are a form of adaptive research design SMARTs never include a control group SMARTs require multiple consents SMARTs are susceptible to high levels of study drop-out

Almirall and many friends Adaptive Interventions April 2016 38 / 66

Current and Future Methodological Work

What am I most excited about lately?!

Current and future study design work: Major surge of interest lately

• n the design of studies to inform cluster-level adaptive interventions

(e.g., staged, multi-level prevention efforts) Future collaborative work: Greater and greater emphasis on real-world aspects of adaptive interventions (e.g., prime for nursing or health services type researchers) Current and future statistical work: Dr. Nahum-Shani and I are currently developing linear mixed models for longitudinal and clustered SMART data. (Hard!)

Almirall and many friends Adaptive Interventions April 2016 39 / 66

Special Issue in Journal of Clinical Child and Adolescent Psychology

Adaptive Interventions in Child and Adolescent Mental Health Editors: Andrea Chronis-Tuscano and Daniel Almirall Foreword: Adaptive interventions in CAMH, literature review, summarizing purpose of the special issue, and looking forward Topics: Over 10 papers covering anxiety, depression, autism, prevention, ADHD, child obesity Discussion: Dr. Joel Sherrill, NIMH Division of Services and Interventions Research, NIMH

Almirall and many friends Adaptive Interventions April 2016 40 / 66

Thank you! Questions?

dalmiral@umich.edu, http://www-personal.umich.edu/∼dalmiral/ Funding for Methods Development NIMH: R03-MH09795401 (PI: Almirall) NIDA: R01-DA039901 (Co-PIs: Almirall and Nahum-Shani) NIDA: P50-DA10075 (PI: Murphy; Co-I: Almirall) Funding for some of the SMARTs Presented Autism Speaks: Grant 5666 (PI: Kasari; Co-I: Almirall) NICHD: R01-HD073975 (PI: Kasari; Co-I: Almirall) IES (PI: Kasari; Co-I: Almirall) NIMH: R01-MH099898 (PI: Kilbourne; Co-I: Almirall)

Almirall and many friends Adaptive Interventions April 2016 41 / 66

Extra, Back-pocket Slides; Slightly More Technical

Almirall and many friends Adaptive Interventions April 2016 42 / 66

Challenges in the Conduct of this Initial Autism SMART

Slow responder rate, while based on prior data, was lower than anticipated during the design of the trial. Responder/Slow-responder measure could be improved to make more useful in actual practice. There was some disconnect with the definition of slow-response status and the therapist’s clinical judgment.

Almirall and many friends Adaptive Interventions April 2016 43 / 66

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2

Almirall and many friends Adaptive Interventions April 2016 44 / 66

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 E[Y (1, 1)] = β0 + β1 + β2 = (JASP,JASP+) E[Y (1, −1)] = β0 + β1 − β2 = (JASP,AAC) E[Y (−1, .)] = β0 − β1 = (AAC,AAC+)

Almirall and many friends Adaptive Interventions April 2016 44 / 66

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 −2β1 + β2 = (AAC,AAC+) vs (JASP,JASP+) −2β1 − β2 = (AAC,AAC+) vs (JASP,AAC) −2β2 = (JASP,AAC) vs (JASP,JASP+)

Almirall and many friends Adaptive Interventions April 2016 45 / 66

How Do We Estimate this Marginal Model?

E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 The observed data is {Xi, A1i, Ri, A2i, Yi}, i = 1, . . . , N. Regressing Y on [1, X, A1, I(A1 = 1)A2] often won’t work. Why? By design, there is an imbalance in the types individuals following AI#1 vs AI#3 (for example)? This imbalance is due to a post-randomization variable R. Adding R to this regression does not fix this and may make it worse! How do we account for the fact that responders to JASP are consistent with two of the embedded AIs? We use something called weighted-and-replicated regression. It is easy!

Almirall and many friends Adaptive Interventions April 2016 46 / 66

Before Weighting-and-Replicating

Almirall and many friends Adaptive Interventions April 2016 47 / 66

After Weighting-and-Replicating

Almirall and many friends Adaptive Interventions April 2016 48 / 66

Weighted-and-Replicated Regression Estimator (WRR)

Statistical foundation found in work by Orellana, Rotnitzky and Robins: Robins JM, Orellana L, Rotnitzky A. Estimation and extrapolation in

• ptimal treatment and testing strategies. Statistics in Medicine. 2008

Jul; 27:4678-4721. Orellana L, Rotnitzky A, Robins JM. Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part I: Main Content. Int J Biostat. 2010; 6(2): Article No. 8. (...ditto...), Part II: Proofs of Results. Int J Biostat. 2010;6(2): Article No. 9. 4678-4721. Very nicely explained and implemented with SMART data in: Nahum-Shani I, Qian M, Almirall D, et al. Experimental design and primary data analysis methods for comparing adaptive interventions. Psychol Methods. 2012 Dec; 17(4): 457-77.

Almirall and many friends Adaptive Interventions April 2016 49 / 66

Weighted-and-Replicated Regression Estimator (WRR)

Weighting (IPTW): By design, each individual/unit has a different probability of following the sequence of treatment s/he was offered (weights account for this)

◮ e.g., W = 2I{A1 = 1, R = 1} + 2I{A1 = −1} + 4I{A1 = 1, R = 0}.

Replication: Some individuals may be consistent with multiple embedded regimes (replication takes advantage of this and permits pooling covariate information)

◮ e.g., Replicate (double) the responders to JASP: assign A2 = 1 to half

and A2 = −1 to the other half

◮ e.g., The new data set is of size M = N + I{A1 = 1, R = 1}

Implementation is as easy as running a weighted least squares: (ˆ η, ˆ β) = arg min

η,β

1 M

M

• i=1

Wi(Yi − µ(Xi, A1i, A2i; η, β))2. SE’s: Use ASEs to account for weighting/replicating (or bootstrap).

Almirall and many friends Adaptive Interventions April 2016 50 / 66

An Interesting Connection Between Estimators

Recall Robins’ G-Computation Estimator (not to be confused with the G-Estimator which is an entirely different thing!:)

• E[Y (1, 1)] =

E[Y |A] Pr[R = 1|JASP] + E[Y |C](1 − Pr[R = 1|JASP])

• E[Y (1, −1)] =

E[Y |A] Pr[R = 1|JASP] + E[Y |B](1 − Pr[R = 1|JASP])

• E[Y (−1, .)] =

E[Y |D] Pr[R = 1|AAC] + E[Y |E](1 − Pr[R = 1|AAC]) This estimator is algebraically identical to fitting the WRR Estimator with no covariates and sample-proportion estimated weights (rather than the known true weights). Comparing these two provides a way to compare the added-value of adjusting for covariates in terms of statistical efficiency gains.

Almirall and many friends Adaptive Interventions April 2016 51 / 66

Results from an Analysis of the Autism SMART

Recall: N = 61, and the primary outcome is SCU at Week 24 (SD=34.6).

WRR with no Covts WRR with Covts and with SAMPLE and Known Wt PROP Wt (G-Comp) ESTIMAND EST SE PVAL EST SE PVAL (AAC,AAC+) 60.5 5.8 < 0.01 61.0 6.0 < 0.01 (JASP,AAC) 42.6 4.9 < 0.01 38.2 6.9 < 0.01 (JASP,JASP+) 36.3 5.0 < 0.01 40.0 8.0 < 0.01 (AAC,AAC+) vs (JASP,JASP+) 24.3 7.9 < 0.01 21.0 10.2 0.04 (AAC,AAC+) vs (JASP,AAC) 17.9 8.2 0.03 22.8 9.4 0.02 (JASP,AAC) vs (JASP,JASP+) 6.4 3.8 0.10

• 1.8

7.7 0.82 What’s the lesson? The regression approach is more useful. (And, it is a good idea to adjust for baseline covariates!) Of course, this is well-known. But the story gets even more interesting...

Almirall and many friends Adaptive Interventions April 2016 52 / 66

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }.

Almirall and many friends Adaptive Interventions April 2016 53 / 66

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }. However, from work by Robins and colleagues (1995; also, Hirano et al (2003)), there are gains in statistical efficiency if using an WRR with weights that are estimated using auxiliary baseline (L1) and time-varying (L2) covariate information. Here’s how to do it with the autism SMART: The observed data is now {L1i, Xi, A1i, Ri, L2i, A2i, Yi} Use logistic regression to get p1 = Pr(A1 | L1, X) Use logistic regression to get p2 = Pr(A2 | L1, X, A1 = 1, R = 0, L2). Use W = I{A1 = 1, R = 0}/( p1 p2) + I{ everyone else }/ p1.

Almirall and many friends Adaptive Interventions April 2016 53 / 66

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }. However, from work by Robins and colleagues (1995; also, Hirano et al (2003)), there are gains in statistical efficiency if using an WRR with weights that are estimated using auxiliary baseline (L1) and time-varying (L2) covariate information. Here’s how to do it with the autism SMART: The observed data is now {L1i, Xi, A1i, Ri, L2i, A2i, Yi} Use logistic regression to get p1 = Pr(A1 | L1, X) Use logistic regression to get p2 = Pr(A2 | L1, X, A1 = 1, R = 0, L2). Use W = I{A1 = 1, R = 0}/( p1 p2) + I{ everyone else }/ p1. The key is to choose Lt’s that are highly correlated with Y !

Almirall and many friends Adaptive Interventions April 2016 53 / 66

Sim: Relative RMSE for (AAC,AAC+) vs (JASP,JASP+)

Almirall and many friends Adaptive Interventions April 2016 54 / 66

Results from an Analysis of the Autism SMART

Recall: N = 61, and the primary outcome is SCU at Week 24 (SD=34.6).

WRR with Covts WRR with Covts and Known Wt and Covt-Est Wt ESTIMAND EST SE PVAL EST SE PVAL (AAC,AAC+) 60.5 5.8 < 0.01 60.2 5.6 < 0.01 (JASP,AAC) 42.6 4.9 < 0.01 43.1 4.5 < 0.01 (JASP,JASP+) 36.3 5.0 < 0.01 35.4 4.4 < 0.01 (AAC,AAC+) vs (JASP,JASP+) 24.3 7.9 < 0.01 24.9 7.4 < 0.01 (AAC,AAC+) vs (JASP,AAC) 17.9 8.2 0.03 17.1 7.9 0.03 (JASP,AAC) vs (JASP,JASP+) 6.4 3.8 0.10 7.7 3.0 0.01 The WRR implementation with covariates and covariate-estimated weights permits us to obtain scientific information from a SMART with less uncertainty.

Almirall and many friends Adaptive Interventions April 2016 55 / 66

Rule-of-thumb concerning which auxiliary variables to use in the WRR for comparing embedded of AIs in a SMART.

Key is to include in Lt variables which are (highly) correlated with Y , even if not of scientific interest. A potentially useful rule-of-thumb (not dogma): Include in L1, all variables that were used to stratify the initial randomization. Include in L2, all variables that were used to stratify the second randomization. Let the science dictate which X’s to include in the final regression model.

◮ e.g., Investigator may be interested in whether baseline levels of spoken

communication moderate the effect of JASP vs JASP+AAC.

◮ Of course: It is possible for X = L1, but not possible for X to include

any post-A1 measures.

Almirall and many friends Adaptive Interventions April 2016 56 / 66

Challenges to Address in Longitudinal Setting

Modeling Considerations: The intermixing of repeated measures and sequential randomizations requires new modeling considerations to account for the fact that embedded AIs will share paths/trajectories at different time points (this is non-trivial) Implications for Interpreting Longitudinal Models: (1) Comparison of slopes is no longer appropriate in many circumstances; (2) Need for new, clinically relevant, easy-to-understand summary measures of the mean trajectories over time Statistical: Develop an estimator that takes advantage of the within person correlation in the outcome over time

Almirall and many friends Adaptive Interventions April 2016 57 / 66

An Example Marginal Model for Longitudinal Outcomes

Yt : # Socially Communicative Utterances at week t. t = 0, 12, 24, 36 The comparison of embedded AIs with longitudinal data arising from a SMART will require longitudinal models that permit deflections in trajectories and respect the fact that some embedded AIs will share paths/trajectories up to the point of randomization. An example is the following piece-wise linear model: E[Yt(a1, a2)|X] = β0 + ηTX + 1t≤12{β1t + β2ta1} + 1t>12{12β1 + 12β2a1 + β3(t − 12) + β4(t − 12)a1 + β5(t − 12)a1a2} where X’s are mean-centered baseline covariates.

Almirall and many friends Adaptive Interventions April 2016 58 / 66

Modeling Considerations

Regime (-1,0): (AAC, AAC+) 12 24 36 t Y

• β0
• slope =

β1 − β2 slope = β3 − β4

Almirall and many friends Adaptive Interventions April 2016 59 / 66

Modeling Considerations

Regime (1,1): (JASP, JASP+) 12 24 36 t Y

• slope =

β1 + β2 slope = β3 + β4 + β5

Almirall and many friends Adaptive Interventions April 2016 60 / 66

Modeling Considerations

Regime (1,-1): (JASP, AAC) 12 24 36 t Y

• slope =

β1 + β2 slope = β3 + β4 − β5

Almirall and many friends Adaptive Interventions April 2016 61 / 66

Implications of New Modeling Considerations for Summarizing each AI

Potential Solution: Summarize each AI by the area under the curve (during an interval chosen by the investigator) Clinical advantage: AUC is easy to understand clinically; it is the average of the primary outcome over a specific interval of time Statistical inference is easy: AUC is linear function of parameters (β’s) in marginal model

Almirall and many friends Adaptive Interventions April 2016 62 / 66

Statistical: WRR Estimator for Longitudinal Outcomes

We use the following estimating equation to estimate marginal model for longitudinal outcomes: 0 = 1 M

M

• i=1

Di(Xi, ¯ Ai)Vi −1Wi(Yi − µi(Xi, ¯ Ai; β, η)), where Yi: a vector of longitudinal outcomes, i.e. (Yi,0, Yi,12, Yi,24, Yi,36)T; µi a vector of corresponding conditional means; Di: the design matrix, i.e.

• ∂µi(Xi, ¯

Ai;β,η) ∂βT

, ∂µi(Xi, ¯

Ai;β,η) ∂ηT

T ; Wi: a diagonal matrix containing inverse probability of following the

• ffered treatment sequence at each time point;

Vi: working covariance matrix for Yi.

Almirall and many friends Adaptive Interventions April 2016 63 / 66

Child Attention Deficit Hyperactivity Disorder (ADHD)

PI: Pelham (FIU) (N = 153; ages 6-12; 8 month study; monthly non-response based on two teacher ratings (ITB < 0.75 and IRS > 1 domain)

Almirall and many friends Adaptive Interventions April 2016 64 / 66

Analysis of Longitudinal Outcomes in the ADHD SMART

Average classroom performance

• ver the school year for each AI

AI Estimate SE (BMD,BMD+) 21.4 0.91 (BMD,BMD+MED) 21.3 0.95 (MED, MED+BMD) 19.2 0.96 (MED, MED+) 19.0 0.85

Almirall and many friends Adaptive Interventions April 2016 65 / 66

Adaptive Implementation Intervention in Mental Health

PI: Kilbourne; Co-I: Almirall (CO/AR/MI; Aim is to improve uptake of psychosocial intervention for mood disorders; primary aim compared initial REP+EF vs REP+EF+IF.)