Adaptive Interventions: What are they? Why do we need them? and How - - PowerPoint PPT Presentation

Adaptive Interventions: What are they? Why do we need them? and How can we study them?

Daniel Almirall, Inbal Nahum-Shani, Susan A. Murphy Survey Research Center, Institute for Social Research University of Michigan October, 2016 Institute for Translational Research in Children’s Mental Health University of Minnesota

Almirall, Nahum-Shani, Murphy Adaptive Interventions October 2016 1 / 66

Outine for the next 3 talks!

1 Introduction to Adaptive Interventions and SMART

Daniel Almirall

2 From Adaptive to “Just in Time” Adaptive Interventions

Inbal “Billie” Nahum-Shani

3 Micro-randomized Trials in Mobile Health

Susan A. Murphy

Almirall, Nahum-Shani, Murphy Adaptive Interventions October 2016 2 / 66

Introduction to Adaptive Interventions and (Cluster-level) SMART Designs

Daniel Almirall Survey Research Center, Institute for Social Research University of Michigan October 7, 2016 Institute for Translational Research in Children’s Mental Health University of Minnesota

Almirall Adaptive Interventions October 2016 3 / 66

I don’t do any of this research by myself.

Colleagues Billie Nahum-Shani, Univ Mich (psychology, behavioral medicine) Susan A. Murphy, Univ Mich (statistician) Connie Kasari, UCLA (autism) Amy Kilbourne, Univ Mich (psychiatry) Ex-students Xi Lu, Google (statistician) Current Students Tim NeCamp, Univ Mich (statistician) And many others...

Almirall Adaptive Interventions October 2016 4 / 66

Outline

What is an Adaptive Intervention? Why do we need Adaptive Interventions? How can we study Adaptive Interventions? Three SMART Case Studies (2 in Autism, 1 in Adult Mood Disorder)

◮ Characterizing Cognition in Non-verbal Individuals with ASD (CCNIA) ◮ Getting SMART about Social & Academic Engagement (ASD Schools) ◮ Adaptive Implementation of Effective Programs (ADEPT, mood dx)

Myths or Misconceptions about Adaptive Interventions and SMARTs

Almirall Adaptive Interventions October 2016 5 / 66

Intervention often entails a sequential, individualized approach whereby treatment is adapted and re-adapted over time

Almirall Adaptive Interventions October 2016 6 / 66

Q: Why do we need sequential, individualized treatments? A: Heterogeneity!

This type of sequential decision-making is necessary when there is high level of individual heterogeneity in response to treatment.

◮ e.g., what works for one individual may not work for another ◮ e.g., what works now may not work later ◮ e.g., what appears not to work in the short-run has positive long-term

consequences, and vice-versa

◮ e.g., adherence, engagement, and burden all have time-changing

courses

Adaptive Interventions help guide this type of individualized, sequential, treatment decision making

Almirall Adaptive Interventions October 2016 7 / 66

What is an Adaptive Intervention?

Almirall Adaptive Interventions October 2016 8 / 66

Definition of an Adaptive Intervention

Sequence of decision rules that guide whether, how, or when (timing), and based on which measures, to alter the dosage (duration, freq or amount), type, or delivery of treatment(s) at decision stages in course of care. Intended to guide the strategies (eg, continue, augment, switch, step-down) leading to individualized treatment. a.k.a. dynamic treatment regimen/regime, adaptive treatment strategy, treatment policy, treatment algorithms Stepped care intervention models are a special case of adaptive interventions

Almirall Adaptive Interventions October 2016 9 / 66

Example of an Adaptive Intervention in Autism

Some Background First... Intervention efforts have overlooked older, school-aged children w/ASD who do not make signif. progress in spoken communication ≥50% of children with autism who received traditional interventions beginning at age 2 remained non-verbal at age 9 years of age. Failure to develop spoken language by age 5 increases likelihood of poor long-term prognosis of adaptive functioning

Almirall Adaptive Interventions October 2016 10 / 66

Example of an Adaptive Intervention in Autism

Some Background First... Intervention efforts have overlooked older, school-aged children w/ASD who do not make signif. progress in spoken communication ≥50% of children with autism who received traditional interventions beginning at age 2 remained non-verbal at age 9 years of age. Failure to develop spoken language by age 5 increases likelihood of poor long-term prognosis of adaptive functioning Evidence Based: Joint Attention, Symbolic Play, Engagement & Regulation (JASPER): two 1-hr sessions/week at clinic Promising: Augmentative, Alternative Communication (AAC) devices

Almirall Adaptive Interventions October 2016 10 / 66

Example of an Adaptive Intervention in Autism

Some Background First... Intervention efforts have overlooked older, school-aged children w/ASD who do not make signif. progress in spoken communication ≥50% of children with autism who received traditional interventions beginning at age 2 remained non-verbal at age 9 years of age. Failure to develop spoken language by age 5 increases likelihood of poor long-term prognosis of adaptive functioning Evidence Based: Joint Attention, Symbolic Play, Engagement & Regulation (JASPER): two 1-hr sessions/week at clinic Promising: Augmentative, Alternative Communication (AAC) devices However, AAC’s potentially costly & not all children may need it.

Almirall Adaptive Interventions October 2016 10 / 66

Example of an Adaptive Intervention in Autism

Some Background First... Intervention efforts have overlooked older, school-aged children w/ASD who do not make signif. progress in spoken communication ≥50% of children with autism who received traditional interventions beginning at age 2 remained non-verbal at age 9 years of age. Failure to develop spoken language by age 5 increases likelihood of poor long-term prognosis of adaptive functioning Evidence Based: Joint Attention, Symbolic Play, Engagement & Regulation (JASPER): two 1-hr sessions/week at clinic Promising: Augmentative, Alternative Communication (AAC) devices However, AAC’s potentially costly & not all children may need it.

◮ Research is limited. Mostly single-subject studies. No rigorous trials.

Motivation for an adaptive intervention involving AAC’s in context

• f JASPER-EMT among older, minimally-verbal children with

autism.

Almirall Adaptive Interventions October 2016 10 / 66

Example of an Adaptive Intervention in Autism

For minimally verbal children with autism spectrum disorder Stage One JASP for 12 weeks; Stage Two At the end of week 12, determine early sign of response:

◮ IF slow responder: Augment JASP with AAC for 12 weeks; ◮ ELSE IF responder: Maintain JASP for 12 weeks.

‐ ‐ ‐ ‐

Almirall Adaptive Interventions October 2016 11 / 66

Example of an Adaptive Intervention in Autism

For minimally verbal children with autism spectrum disorder Stage One JASP for 12 weeks; Stage Two At the end of week 12, determine early sign of response:

◮ IF slow responder: Augment JASP with AAC for 12 weeks; ◮ ELSE IF responder: Maintain JASP for 12 weeks.

Continue: JASP Responders JASP Augment: JASP + AAC Slow Responders

First‐stage Treatment (Weeks 1‐12) Second‐stage Treatment (Weeks 13‐24) End of Week 12 Responder Status

Almirall Adaptive Interventions October 2016 11 / 66

How was response/slow-response defined?

Percent change from baseline to week 12 was calculated for 7 variables: 7 variables: socially communicative utterances (SCU), percent SCU, mean length utterance, total word roots, words per minute, total comments, unique word combinations Fast Responder: if ≥25% change on 7 measures; Slower Responder: otherwise (this includes kids with no improvement, which is rare)

Almirall Adaptive Interventions October 2016 12 / 66

But how did we come up with this adaptive intervention? Is it a good one? Do we really know how best to sequence and individualize treatments?

Almirall Adaptive Interventions October 2016 13 / 66

Many Unanswered Questions when Building a High-Quality Adaptive Intervention.

Is it better to provide AAC from the start? Who benefits from initial AAC versus who benefits from delayed AAC? For slow responders, what is the effect of providing the AAC vs intensifying JASP (not providing AAC)? Why begin treatment with JASP? Why not a high-quality course of core-principles of discrete trials training? Should we train parents following an initial successful course of treatment? If so, which families benefit most from this?

Almirall Adaptive Interventions October 2016 14 / 66

Insufficient Empirical Evidence or Theories to Address such Questions

In the past we have relied on Expert opinion Clinical expertise Piecing together an adaptive intevrention using results from separate RCTs Our group at Michigan has been developing novel randomized experimental design methods for answering many of these questions. Sequential Multiple Assignment Randomized Trials (SMARTs) address such questions empirically, using experimental design principles.

Almirall Adaptive Interventions October 2016 15 / 66

Sequential Multiple Assignment Randomized Trials (SMART)

Almirall Adaptive Interventions October 2016 16 / 66

What is a Sequential Multiple Assignment Randomized Trial (SMART)?

A type of multi-stage randomized trial design. At each stage, subjects randomized to a set of feasible/ethical treatment options. Treatment options at latter stages may be restricted by response to earlier treatments.

Almirall Adaptive Interventions October 2016 17 / 66

What is a Sequential Multiple Assignment Randomized Trial (SMART)?

A type of multi-stage randomized trial design. At each stage, subjects randomized to a set of feasible/ethical treatment options. Treatment options at latter stages may be restricted by response to earlier treatments. SMARTs were developed explicitly for the purpose of building a high-quality Adaptive Intervention.

Almirall Adaptive Interventions October 2016 17 / 66

SMART Case Study #1: Characterizing Cognition in Non-verbal Individuals with ASD (CCNIA)

Almirall Adaptive Interventions October 2016 18 / 66

Example of a (first-ever) SMART in Autism Research

PI: Kasari (UCLA)

The population of interest: Children with autism spectrum disorder Age: 5-8 Minimally verbal: <20 spontaneous words in a 20-min. language test History of treatment: ≥2 years of prior intervention Functioning: ≥2 year-old on non-verbal intelligence tests

Almirall Adaptive Interventions October 2016 19 / 66

Example of a SMART in Autism Research (N = 61)

PI: Kasari (UCLA)

Almirall Adaptive Interventions October 2016 20 / 66

There are 3 AIs Embedded in this Example Autism SMART

(JASP,JASP+) Subgroups A+C (JASP,AAC) Subgroups A+B (AAC,AAC+) Subgroups D+E

Almirall Adaptive Interventions October 2016 21 / 66

SMARTs permit scientists to answer many interesting questions for building a high-quality adaptive intervention

Primary Aim: What is the best first-stage treatment in terms of spoken communication at week 24: JASP alone vs JASP+AAC?

◮ Kasari, C., et al. (2014). Communication Interventions for Minimally

Verbal Children with Autism: Sequential Multiple Assignment Randomized Trial. Journal of the American Academy of Child and Adolescent Psychiatry.

Almirall Adaptive Interventions October 2016 22 / 66

SMARTs permit scientists to answer many interesting questions for building a high-quality adaptive intervention

Primary Aim: What is the best first-stage treatment in terms of spoken communication at week 24: JASP alone vs JASP+AAC?

◮ Kasari, C., et al. (2014). Communication Interventions for Minimally

Verbal Children with Autism: Sequential Multiple Assignment Randomized Trial. Journal of the American Academy of Child and Adolescent Psychiatry.

Secondary Aim: Compare longitudinal outcomes between the three adaptive interventions embedded in this SMART. Today, I will show you results for the Secondary Aim.

Almirall Adaptive Interventions October 2016 22 / 66

Longitudinal Outcomes in this Example SMART in Autism

Outcomes collected at baseline, and weeks 12, 24 and 36 Primary outcome (verbal): from 20-minute “natural language sample”: Total spontaneous communicative utterances (TSCU) Secondary outcome (non-linguistic): Initiating joint attention (IJA; e.g., pointing; JASP mechanism)

Almirall Adaptive Interventions October 2016 23 / 66

Results

Adaptive (a) TSCU (b) IJA Intervention AUC 95% CI AUC 95%CI (AAC,AAC+) 51.7 [43, 60] 9.5 [7.2,11.8] (JASP,AAC) 36.0 [28, 44] 7.2 [5.6,8.8] (JASP,JASP+) 33.1 [25, 42] 6.6 [5,8.2] No diff null p < 0.01 p < 0.05

Almirall Adaptive Interventions October 2016 24 / 66

Two More SMART Case Studies

Almirall Adaptive Interventions October 2016 25 / 66

SMART Case Study #2: Getting SMART about Social and Academic Engagement in Elementary Aged Children with ASD

Almirall Adaptive Interventions October 2016 26 / 66

Social & Academic Engagement in School-Children w/ASD

Background

Targeting students with ASD, ages 5-12, ≥ 70 IQ In inclusive schools/classrooms in the LAUSD Improve social skills, behavioral, and academic engagement outcomes Early on, social skills interventions rarely studied in the child’s natural environment (befuddles me!); so had no “generalization” Others that were in the child’s natural environment, such as 1:1 pairing with an aide at the school, turned out to have iatrogenic effects!

Social & Academic Engagement in School-Children w/ASD

Set-up and Questions

Evidence Base: Remaking Recess (school-level), Classroom Supports (class-level), Parent-assisted social skills intervention (individual-level at home), Peer-mediated social skills intervention (individual-level) But no empirically derived guidelines to help guide intervention decision making of children with ASD in schools. Combined Peer + Parent is also promising, but this cannot be provided to all children due to cost and not all children need it Begin individual level interventions with Peer or with Parent?

Academic & Social Engagement in School-Children w/ASD

PIs: Kasari; Co-I: Almirall; IES-funded Pilot SMART

This is a 2-arm RCT. But my mad scientist friend had some other scientific questions, leading to a different design...

Social & Academic Engagement in School-Children w/ASD

Set-up and Questions

Evidence Base: Remaking Recess (school-level), Classroom Supports (class-level), Parent-assisted social skills intervention (individual-level at home), Peer-mediated social skills intervention (individual-level) But no empirically derived guidelines to help guide intervention decision making of children with ASD in schools. Begin individual level interventions with Peer or with Parent? Do all classrooms require Classroom Supports? Synergistic effect between CS and individual-level interv.?

Academic & Social Engagement in School-Children w/ASD

PIs: Kasari; Co-I: Almirall; IES-funded Pilot SMART

Primary and Secondary Aims of this Pilot SMART

Primary Aim: Address feasibility and acceptability concerns related to the embedded adaptive interventions

◮ identifying children as early vs. slower responders by the

paraprofessionals in the context of RR,

◮ transitioning children to Parent or Peer at wk8, ◮ providing augmented Peer+Parent to slower responders ◮ not providing augmented treatment to early responders at wk20 ◮ satisfaction with txt sequences by children, parents, teachers,

paraprofessionals & school champions

◮ teacher-rated measures of progress during CS for deciding Parent vs

Peer

Secondary Aim: To obtain preliminary data to support a full-scale SMART.

Almirall Adaptive Interventions October 2016 32 / 66

SMART Case Study #3: Adaptive Implementation of Effective Programs (ADEPT)

Almirall Adaptive Interventions October 2016 33 / 66

Adaptive Implementation Intervention in Mental Health

PI: Kilbourne; Co-I: Almirall (CO/AR/MI; Aim is to improve uptake of psychosocial intervention for mood disorders; primary aim compared initial REP+EF vs REP+EF+IF.)

Non-responding site if: < 50% of previously identified patients receiving at least three LG sessions (3 out of 6)

Summary of the Talk

You learned about Adaptive Interventions You learned about SMART study designs for developing them You learned about 3 SMARTs in the field Two of them were cluster-randomized SMARTs used to answer critical questions in the development of cluster-level adaptive interventions. We have sample size/power calculators for comparing the mean

• utcome between two embedded adaptive interventions using SMART

data (joint work with Tim NeCamp).

Myths and Misconceptions about Adaptive Interventions and SMARTs

Almirall Adaptive Interventions October 2016 36 / 66

Myths and Misconceptions about Adaptive Interventions

Tailoring variables cannot differ based on previous intervention An adaptive intervention must recommend a single intervention component at each decision point Adaptive interventions seek to replace clinical judgement Adaptive interventions are only relevant in treatment settings Adaptive interventions are non-standard because they involve randomization

Almirall Adaptive Interventions October 2016 37 / 66

Myths and Misconceptions about SMART Studies

SMARTs require prohibitively large sample sizes All SMARTs require Multiple-Comparisons Adjustments All research on adaptive interventions requires a SMART All SMARTs must include an embedded tailoring variable All aspects of an embedded adaptive intervention must be randomized SMARTs are a form of adaptive research design SMARTs never include a control group SMARTs require multiple consents SMARTs are susceptible to high levels of study drop-out

Almirall Adaptive Interventions October 2016 38 / 66

Special Issue in Journal of Clinical Child and Adolescent Psychology APA’s Division 53 Journal

Adaptive Interventions in Child and Adolescent Mental Health Editors: Daniel Almirall and Andrea Chronis-Tuscano Topics: Over 10 blinded, externally peer-reviewed papers covering anxiety, depression, autism, prevention, ADHD, child obesity Discussion: Dr. Joel Sherrill, NIMH Division of Services and Interventions Research, NIMH

Almirall Adaptive Interventions October 2016 39 / 66

Thank you! Questions?

dalmiral@umich.edu, http://www-personal.umich.edu/∼dalmiral/ Funding for Methods Development NIMH: R03-MH09795401 (PI: Almirall) NIDA: R01-DA039901 (Co-PIs: Almirall and Nahum-Shani) NIDA: P50-DA10075 (PI: Murphy; Co-I: Almirall) Funding for some of the SMARTs Presented Autism Speaks: Grant 5666 (PI: Kasari; Co-I: Almirall) NICHD: R01-HD073975 (PI: Kasari; Co-I: Almirall) IES (PI: Kasari; Co-I: Almirall) NIMH: R01-MH099898 (PI: Kilbourne; Co-I: Almirall)

Almirall Adaptive Interventions October 2016 40 / 66

Extra, Back-pocket Slides; Slightly More Technical

Almirall Adaptive Interventions October 2016 41 / 66

Definition of an Adaptive Intervention, in symbols

{S1, a1, S2(a1), a2, . . . , ST(¯ aT−1), aT} St is the state or status of the individual/unit at time t and at indexes a possible action (treatment) at time t

◮ e.g., intensify medication dose? ◮ e.g., add medication to behavioral intervention? ◮ e.g., continue treatment and monitor?

An adaptive intervention is a sequence of decision rules {d1(s1), d2(s1, a1, s2), . . . , dT(¯ aT−1, ¯ sT)}.

Almirall Adaptive Interventions October 2016 42 / 66

Interventions for Minimally Verbal Children with Autism

PIs: Kasari(UCLA), Almirall(Mich), Kaiser(Vanderbilt), Smith(Rochester), Lord(Cornell)

Primary and Secondary Aims

Primary Aim: What is the best first-stage treatment in terms of spoken communication at week 24: JASP vs DTT? (Sized N = 192 for this aim; compares A+B+C+D vs E+F+G+H) Secondary Aim 1: Determine whether adding a parent training provides additional benefit among children who demonstrate a positive early response to either JASP or DTT (D+H vs C+G). Secondary Aim 2: Determine whether adding JASP+DTT provides additional benefit among children who demonstrate a slow early response to either JASP or DTT (A+E vs B+F). Secondary Aim 3: Compare eight pre-specified adaptive interventions. [Note, we can now compare always JASP vs always DTT!]

Almirall Adaptive Interventions October 2016 44 / 66

Challenges in the Conduct of this Initial Autism SMART

Slow responder rate, while based on prior data, was lower than anticipated during the design of the trial. Responder/Slow-responder measure could be improved to make more useful in actual practice. There was some disconnect with the definition of slow-response status and the therapist’s clinical judgment.

Almirall Adaptive Interventions October 2016 45 / 66

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2

Almirall Adaptive Interventions October 2016 46 / 66

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 E[Y (1, 1)] = β0 + β1 + β2 = (JASP,JASP+) E[Y (1, −1)] = β0 + β1 − β2 = (JASP,AAC) E[Y (−1, .)] = β0 − β1 = (AAC,AAC+)

Almirall Adaptive Interventions October 2016 46 / 66

A Simple Regression Model for Comparing the Embedded AIs

Y (a1, a2) denotes SCU at Wk 24 under AI (a1, a2). X’s are mean-centered baseline (pre-txt) covariates. Consider the following marginal model: E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 −2β1 + β2 = (AAC,AAC+) vs (JASP,JASP+) −2β1 − β2 = (AAC,AAC+) vs (JASP,AAC) −2β2 = (JASP,AAC) vs (JASP,JASP+)

Almirall Adaptive Interventions October 2016 47 / 66

How Do We Estimate this Marginal Model?

E[Y (a1, a2)|X] = β0 + ηTX + β1a1 + β2I(a1 = 1)a2 The observed data is {Xi, A1i, Ri, A2i, Yi}, i = 1, . . . , N. Regressing Y on [1, X, A1, I(A1 = 1)A2] often won’t work. Why? By design, there is an imbalance in the types individuals following AI#1 vs AI#3 (for example)? This imbalance is due to a post-randomization variable R. Adding R to this regression does not fix this and may make it worse! How do we account for the fact that responders to JASP are consistent with two of the embedded AIs? We use something called weighted-and-replicated regression. It is easy!

Almirall Adaptive Interventions October 2016 48 / 66

Before Weighting-and-Replicating

Almirall Adaptive Interventions October 2016 49 / 66

After Weighting-and-Replicating

Almirall Adaptive Interventions October 2016 50 / 66

Weighted-and-Replicated Regression Estimator (WRR)

Statistical foundation found in work by Orellana, Rotnitzky and Robins: Robins JM, Orellana L, Rotnitzky A. Estimation and extrapolation in

• ptimal treatment and testing strategies. Statistics in Medicine. 2008

Jul; 27:4678-4721. Orellana L, Rotnitzky A, Robins JM. Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part I: Main Content. Int J Biostat. 2010; 6(2): Article No. 8. (...ditto...), Part II: Proofs of Results. Int J Biostat. 2010;6(2): Article No. 9. 4678-4721. Very nicely explained and implemented with SMART data in: Nahum-Shani I, Qian M, Almirall D, et al. Experimental design and primary data analysis methods for comparing adaptive interventions. Psychol Methods. 2012 Dec; 17(4): 457-77.

Almirall Adaptive Interventions October 2016 51 / 66

Weighted-and-Replicated Regression Estimator (WRR)

Weighting (IPTW): By design, each individual/unit has a different probability of following the sequence of treatment s/he was offered (weights account for this)

◮ e.g., W = 2I{A1 = 1, R = 1} + 2I{A1 = −1} + 4I{A1 = 1, R = 0}.

Replication: Some individuals may be consistent with multiple embedded regimes (replication takes advantage of this and permits pooling covariate information)

◮ e.g., Replicate (double) the responders to JASP: assign A2 = 1 to half

and A2 = −1 to the other half

◮ e.g., The new data set is of size M = N + I{A1 = 1, R = 1}

Implementation is as easy as running a weighted least squares: (ˆ η, ˆ β) = arg min

η,β

1 M

M

• i=1

Wi(Yi − µ(Xi, A1i, A2i; η, β))2. SE’s: Use ASEs to account for weighting/replicating (or bootstrap).

Almirall Adaptive Interventions October 2016 52 / 66

An Interesting Connection Between Estimators

Recall Robins’ G-Computation Estimator (not to be confused with the G-Estimator which is an entirely different thing!:)

• E[Y (1, 1)] =

E[Y |A] Pr[R = 1|JASP] + E[Y |C](1 − Pr[R = 1|JASP])

• E[Y (1, −1)] =

E[Y |A] Pr[R = 1|JASP] + E[Y |B](1 − Pr[R = 1|JASP])

• E[Y (−1, .)] =

E[Y |D] Pr[R = 1|AAC] + E[Y |E](1 − Pr[R = 1|AAC]) This estimator is algebraically identical to fitting the WRR Estimator with no covariates and sample-proportion estimated weights (rather than the known true weights). Comparing these two provides a way to compare the added-value of adjusting for covariates in terms of statistical efficiency gains.

Almirall Adaptive Interventions October 2016 53 / 66

Results from an Analysis of the Autism SMART

Recall: N = 61, and the primary outcome is SCU at Week 24 (SD=34.6).

WRR with no Covts WRR with Covts and with SAMPLE and Known Wt PROP Wt (G-Comp) ESTIMAND EST SE PVAL EST SE PVAL (AAC,AAC+) 60.5 5.8 < 0.01 61.0 6.0 < 0.01 (JASP,AAC) 42.6 4.9 < 0.01 38.2 6.9 < 0.01 (JASP,JASP+) 36.3 5.0 < 0.01 40.0 8.0 < 0.01 (AAC,AAC+) vs (JASP,JASP+) 24.3 7.9 < 0.01 21.0 10.2 0.04 (AAC,AAC+) vs (JASP,AAC) 17.9 8.2 0.03 22.8 9.4 0.02 (JASP,AAC) vs (JASP,JASP+) 6.4 3.8 0.10

• 1.8

7.7 0.82 What’s the lesson? The regression approach is more useful. (And, it is a good idea to adjust for baseline covariates!) Of course, this is well-known. But the story gets even more interesting...

Almirall Adaptive Interventions October 2016 54 / 66

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }.

Almirall Adaptive Interventions October 2016 55 / 66

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }. However, from work by Robins and colleagues (1995; also, Hirano et al (2003)), there are gains in statistical efficiency if using an WRR with weights that are estimated using auxiliary baseline (L1) and time-varying (L2) covariate information. Here’s how to do it with the autism SMART: The observed data is now {L1i, Xi, A1i, Ri, L2i, A2i, Yi} Use logistic regression to get p1 = Pr(A1 | L1, X) Use logistic regression to get p2 = Pr(A2 | L1, X, A1 = 1, R = 0, L2). Use W = I{A1 = 1, R = 0}/( p1 p2) + I{ everyone else }/ p1.

Almirall Adaptive Interventions October 2016 55 / 66

Improving the Efficiency of the WRR by Estimating the Known Weights with Covariates

By design, we know the true weights. That is, Since Pr(A1) = 1/2 and Pr(A2 = 1 | A1 = 1, R = 0) = 1/2, then W = 4I{A1 = 1, R = 0} + 2I{ everyone else }. However, from work by Robins and colleagues (1995; also, Hirano et al (2003)), there are gains in statistical efficiency if using an WRR with weights that are estimated using auxiliary baseline (L1) and time-varying (L2) covariate information. Here’s how to do it with the autism SMART: The observed data is now {L1i, Xi, A1i, Ri, L2i, A2i, Yi} Use logistic regression to get p1 = Pr(A1 | L1, X) Use logistic regression to get p2 = Pr(A2 | L1, X, A1 = 1, R = 0, L2). Use W = I{A1 = 1, R = 0}/( p1 p2) + I{ everyone else }/ p1. The key is to choose Lt’s that are highly correlated with Y !

Almirall Adaptive Interventions October 2016 55 / 66

Sim: Relative RMSE for (AAC,AAC+) vs (JASP,JASP+)

Almirall Adaptive Interventions October 2016 56 / 66

Results from an Analysis of the Autism SMART

Recall: N = 61, and the primary outcome is SCU at Week 24 (SD=34.6).

WRR with Covts WRR with Covts and Known Wt and Covt-Est Wt ESTIMAND EST SE PVAL EST SE PVAL (AAC,AAC+) 60.5 5.8 < 0.01 60.2 5.6 < 0.01 (JASP,AAC) 42.6 4.9 < 0.01 43.1 4.5 < 0.01 (JASP,JASP+) 36.3 5.0 < 0.01 35.4 4.4 < 0.01 (AAC,AAC+) vs (JASP,JASP+) 24.3 7.9 < 0.01 24.9 7.4 < 0.01 (AAC,AAC+) vs (JASP,AAC) 17.9 8.2 0.03 17.1 7.9 0.03 (JASP,AAC) vs (JASP,JASP+) 6.4 3.8 0.10 7.7 3.0 0.01 The WRR implementation with covariates and covariate-estimated weights permits us to obtain scientific information from a SMART with less uncertainty.

Almirall Adaptive Interventions October 2016 57 / 66

Rule-of-thumb concerning which auxiliary variables to use in the WRR for comparing embedded of AIs in a SMART.

Key is to include in Lt variables which are (highly) correlated with Y , even if not of scientific interest. A potentially useful rule-of-thumb (not dogma): Include in L1, all variables that were used to stratify the initial randomization. Include in L2, all variables that were used to stratify the second randomization. Let the science dictate which X’s to include in the final regression model.

◮ e.g., Investigator may be interested in whether baseline levels of spoken

communication moderate the effect of JASP vs JASP+AAC.

◮ Of course: It is possible for X = L1, but not possible for X to include

any post-A1 measures.

Almirall Adaptive Interventions October 2016 58 / 66

Challenges to Address in Longitudinal Setting

Modeling Considerations: The intermixing of repeated measures and sequential randomizations requires new modeling considerations to account for the fact that embedded AIs will share paths/trajectories at different time points (this is non-trivial) Implications for Interpreting Longitudinal Models: (1) Comparison of slopes is no longer appropriate in many circumstances; (2) Need for new, clinically relevant, easy-to-understand summary measures of the mean trajectories over time Statistical: Develop an estimator that takes advantage of the within person correlation in the outcome over time

Almirall Adaptive Interventions October 2016 59 / 66

An Example Marginal Model for Longitudinal Outcomes

Yt : # Socially Communicative Utterances at week t. t = 0, 12, 24, 36 The comparison of embedded AIs with longitudinal data arising from a SMART will require longitudinal models that permit deflections in trajectories and respect the fact that some embedded AIs will share paths/trajectories up to the point of randomization. An example is the following piece-wise linear model: E[Yt(a1, a2)|X] = β0 + ηTX + 1t≤12{β1t + β2ta1} + 1t>12{12β1 + 12β2a1 + β3(t − 12) + β4(t − 12)a1 + β5(t − 12)a1a2} where X’s are mean-centered baseline covariates.

Almirall Adaptive Interventions October 2016 60 / 66

Modeling Considerations

Regime (-1,0): (AAC, AAC+) 12 24 36 t Y

• β0
• slope =

β1 − β2 slope = β3 − β4

Almirall Adaptive Interventions October 2016 61 / 66

Modeling Considerations

Regime (1,1): (JASP, JASP+) 12 24 36 t Y

• slope =

β1 + β2 slope = β3 + β4 + β5

Almirall Adaptive Interventions October 2016 62 / 66

Modeling Considerations

Regime (1,-1): (JASP, AAC) 12 24 36 t Y

• slope =

β1 + β2 slope = β3 + β4 − β5

Almirall Adaptive Interventions October 2016 63 / 66

Implications of New Modeling Considerations for Summarizing each AI

Potential Solution: Summarize each AI by the area under the curve (during an interval chosen by the investigator) Clinical advantage: AUC is easy to understand clinically; it is the average of the primary outcome over a specific interval of time Statistical inference is easy: AUC is linear function of parameters (β’s) in marginal model

Almirall Adaptive Interventions October 2016 64 / 66

Child Attention Deficit Hyperactivity Disorder (ADHD)

PI: Pelham (FIU) (N = 153; ages 6-12; 8 month study; monthly non-response based on two teacher ratings (ITB < 0.75 and IRS > 1 domain)

Almirall Adaptive Interventions October 2016 65 / 66

Analysis of Longitudinal Outcomes in the ADHD SMART

Average classroom performance

• ver the school year for each AI

AI Estimate SE (BMD,BMD+) 21.4 0.91 (BMD,BMD+MED) 21.3 0.95 (MED, MED+BMD) 19.2 0.96 (MED, MED+) 19.0 0.85

Almirall Adaptive Interventions October 2016 66 / 66