Estimating beyond the trial-represented population by incorporating - - PowerPoint PPT Presentation

Estimating beyond the trial-represented population by incorporating studies with self-selected treatments

Eloise E. Kaizar

Department of Statistics Ohio State University

February 26, 2016

Joint work with: Joel Greenhouse, Kelly Kelleher, Taylor Pressler Vydra, Howard Seltman

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 1 / 18

Does antidepressant use cause suicide among adolescents?

No single study could definitively answer Available randomized evidence (Hammad, et al. 2006)

24 studies of various psychological disorders 4582 subjects

No completed suicides

Instead adjudicated “suicidality events" n=87 (2%) experienced suicidality

Goal: Estimate the ’average’ safety of antidepressant use

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 2 / 18

Outline

1

Are Studies Generalizable?

2

Combining RCT and Observational Data

3

Extensions

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 3 / 18

Are Studies Generalizable?

Who do RCTs actually study?

Studies consistently show that it is usual for RCTs to exclude half

• r more of the population of interest due to eligibility criteria

Asthma, > 57% (Travers, et al, 2007) Alcohol treatment, 6 − 70% (Humphreys and Weisner, 2000) Antidepressants, mean=66% (Zimmerman, et al, 2004)

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 4 / 18

Are Studies Generalizable?

Assessing Consistency with Target Populations

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 5 / 18

Are Studies Generalizable?

Assessing Consistency with Target Populations, cont.

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 6 / 18

Combining RCT and Observational Data

General Methods for Combining RCT and Observational Data

Three-level meta-analysis (Prevost, Abrams & Jones, 2000) Response surface (Rubin, 1990) Bias-adjusted model, e.g.,

Confidence Profile Method (Eddy, Hasselblad & Shachter, 1990) Multiple bias models (Greenland, 2005) Cross Design Synthesis (US GAO, 1992)

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 7 / 18

Combining RCT and Observational Data

Cross Design Synthesis

Treatment effect External validity moderators

Subjects select treatment Subjects randomized to treatment

E x t e r n a l v a l i d i t y m

• d

e r a t

• r

s I n t e r n a l v a l i d i t y m

• d

e r a t

• r

s Treatment effect

a b

Nonrandomized Randomized

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 8 / 18

Combining RCT and Observational Data

Framework

Use RCT to estimate effect size in the group it represents

Rely on strong internal validity

Use observational data to estimate the generalizability bias

Rely on strong external validity

Put the two estimators together to estimate the population average effect size

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 9 / 18

Combining RCT and Observational Data

Simplest Cross Design Synthesis

Linear Generalizability Bias

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 10 / 18

Combining RCT and Observational Data

Simple Example: Insulin Pump Use

Question: Is insulin pump use on average effective in improving metabolic control in the total population of diabetic patients?

Outcome: Mean A1C level (lower is better) Control: Insulin injections

Goal: Estimate the average treatment effect for use in policy decision making Issue: RCTs exclude the noncompliant (< 4 checks per day) Doyle, et al (2004), Paris, et al (2009)

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 11 / 18

Combining RCT and Observational Data (Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 12 / 18

Combining RCT and Observational Data

Assumptions for Unbiasedness

No confounding within the levels of inclusion U ⊥ ⊥ T|X or E(Y|T, U, X) = g(T, X)

Socioeconomic status is independent of the choice to use an insulin pump, within monitoring groups, OR Socioeconomic status does not influence metabolic control.

Confounding due to inclusion criteria and other variables are separate U ⊥ ⊥ X|T and U ⊥ ⊥ X and E(Y|T, U, X) = g1(T, X) + g2(T, U)

The frequency of glucose monitoring is independent of socioeconomic status and does not moderate its effect on metabolic control. Note that unlike the experimental estimator, the inclusion criterion can moderate treatment, as long as this is separate from the other variables.

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 13 / 18

Combining RCT and Observational Data

CDS Properties

Bias

CDS is better whenever the scaled difference in expected treatment selection error is smaller than the difference in generalizability bias (sample selection error)

Variance

CDS is never better, although the difference is expected to be small whenever the observational data is large

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 14 / 18

Extensions

Improving the Plausibility of Assumption 1

Assumption 1: No confounding within the levels of inclusion Approach: Use existing methods (e.g., propensity scores) to reduce/eliminate confounding within levels of inclusion Simulation

Three standard normal covariates, constant correlation One influences treatment assignment Another influences exclusion.

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 15 / 18

Extensions

Estimating Generalizability Bias

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 16 / 18

Summary

Summary

CDS offers a framework for thinking about estimating average treatment effects in populations of interest that are not represented by RCTs. Simple CDS models rely on two rather restrictive (relative to ideal) assumptions. More work is necessary to relax such assumptions and improve its usefulness. CDS often replaces one data limitation with several others:

Data availability (including treatment in the observational study) Data harmonization

(Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 17 / 18

References

References

Doyle E A, Weinzimer SA, Steffen AT, Ahern JAH, Vincent M, Tamborlane WV. 2004. A randomized, prospective trial comparing the efficacy of continuous subcutaneous insulin infusion with multiple daily injections using insulin glargine. Diabetes Care. 27: 1554-1558. Eddy DM, Hasselblad V, Shachter R. 1990. A Bayesian method for synthesizing evidence: the confidence profile method. International Journal of Technology Assessment in Health Care. 6: 31-56. Greenland S. Multiple-bias modelling for analysis of observational data. 2005. Journal of the Royal Statistical Society: Series A. 168: 267-306. Kaizar EE. 2011. Estimating treatment effect via simple cross design synthesis. Statistics in Medicine. 30(25):2986-3009. Kaizar EE. 2015. Incorporating Both Randomized and Observational Data into a Single Analysis. Annual Review of Statistics and Its Application. 2:49-72 Paris CA, Imperatore G, Klingensmith G, Petitti D, Rodriguez B, Anderson AM, Schwartz ID, Standiford DA, Pihoker C.

• 2009. Predictors of insulin regimens and impact on outcomes in youth with type 1 diabetes: the SEARCH for Diabetes in

Youth study. Journal of Pediatrics. 155: 183-189. Pressler T and Kaizar EE, 2013. The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias. Statistics in Medicine. 32(20):3552-3568. Prevost TC, Abrams, KR, Jones DR. 2000. Hierarchical models in generalized synthesis of evidence: an example based

• n studies of breast cancer screening. Statistics in Medicine. 19: 3359-3376.

Rubin, D. 1990. A new perspective on meta-analysis. In K. M. Wachter & M. L. Straff, eds., The Future of Meta-Analysis. Russel Sage Foundation, 155-165. U.S. General Accounting Office. 1992. Cross-design synthesis: A new strategy for medical effectiveness research (GAO/PEMD-92-18). U.S. General Accounting Office, Washington, DC. http://archive.gao.gov/d31t10/145906.pdf. (Ohio State University) Estimating Beyond the Trial Ross-Royall Symposium 18 / 18