Using Different Comparison Group Selection Methodologies to Evaluate the Impact of Health Care Innovation June 25, 2017 #ARM17 @echo_yliu Yiyan (Echo) Liu, PhD, and Emily Gillen, PhD www.rti.org RTI International is a registered trademark and a trade name of Research Triangle Institute.
Acknowledgment and Disclaimer This work was supported through a contract from the Centers for Medicare and Medicaid Innovation within the Centers for Medicare & Medicaid Services (HHSM-500-2010-00021I). The contents of this presentation are solely the responsibility of the authors and do not necessarily represent the official views of the U.S. Department of Health and Human Services or any of its agencies. 2
Road Map Introduction – Comparison group selection – Health Care Innovation Awards Objective Comparison Group Selection – Matching on the propensity score – Inverse probability of treatment weighting (IPTW) using the propensity score – Entropy balancing weighting Model Covariates Balance Check Difference-in-differences (DinD) Regression Results 3
Comparison Groups in Observational Studies Observational studies are increasingly common in the social sciences. – Randomization is not always possible. Unethical/not feasible Evaluations designed after an intervention has been under way (no comparison group designated) Evaluations require a comparison group. – If the treatment group is selected based on eligibility criteria and the comparison group is created based on similar, but not eligible, individuals, we could be introducing bias into the evaluation (if eligibility criteria are associated with the outcome). Could create even more bias if the treatment group is selected based on eligibility criteria and the comparison group includes eligible individuals who opted out of enrollment (if enrollment is associated with the outcome). 4
Comparison Group Selection The conditional probability of receiving a treatment given a set of observable characteristics (Rosenbaum and Rubin,1983) – A method to reduce bias in comparison group creation Using propensity scores (Austin, 2009) – Stratification Divide the sample into strata (e.g., quintiles or deciles), and use propensity scores to match treatment and comparison observations within strata. – Covariate adjustment Propensity score is used as a covariate in the outcome regression model. 5
Comparison Group Selection (continued) Using propensity scores (continued): – Matching Match treatment and comparison group observations on their propensity scores. Some examples/decisions to make o Selection of match: nearest neighbor or greedy o How many matches: one-to-one or one-to-many o Replacement: with or without – Weighting (inverse probability of treatment weighting) Individuals in the treatment group receive a weight of 1; the comparison group receives a weight of the following: 𝑄𝑠𝑝𝑞𝑓𝑜𝑡𝑗𝑢𝑧 𝑇𝑑𝑝𝑠𝑓 1 − 𝑄𝑠𝑝𝑞𝑓𝑜𝑡𝑗𝑢𝑧 𝑇𝑑𝑝𝑠𝑓 Entropy balancing weighting (Hainmueller, 2012) Adjusts inequalities in representation with respect to the first, second, and possibly higher moments of the covariate distributions. 6
Concerns with Comparison Group Achieving balance – Calculate the absolute standardized differences between the treatment and comparison groups (before propensity score methodology applied and after). – Check whether the absolute standardized difference decreases and whether it achieves acceptable balance. One frame of reference: many researchers consider that an absolute standardized difference ≤ 0.10 indicates acceptable balance (Austin, 2011a). Sample size – Matching (especially 1-to-1) can lead to a loss of observations. If you don’t have an adequate match in the prospective comparison group, you may need to drop treatment group observations. Can be a problem with small samples 7
Health Care Innovation Awards (HCIA) HCIA: Up to $1 billion in awards to organizations with creative ideas to make improvements in delivery systems, health outcomes, or the cost of care. Data from one HCIA awardee – Time span: April 2011 – December 2015 – Fee-for-service (FFS) Medicare claims – Treatment group: 6,476 Medicare FFS beneficiaries – Spending and utilization outcomes Medicare health care spending (Parts A and B) All-cause hospital inpatient admissions Emergency department (ED) visits not leading to a hospitalization 8
Objective Because no gold standard in applying propensity score methods, – How do researchers choose which method to use? – What effect does that choice have? We use the HCIA awardee data to compare the following: 1. Propensity score matching 2. Propensity score weighting 3. Entropy balancing weighting We will apply these methods to construct different comparison groups, and then look at the innovation effect on three outcomes: – Medicare spending – Inpatient admissions – ED visits 9
Propensity Score Model Covariates Apply the same inclusion and exclusion criteria used for the treatment group to the initial pool of individuals eligible for the comparison group, to the extent possible – Geographic region – Age restrictions Define model covariates: demographics, health conditions, eligibility criteria, and payments and utilization in the period prior to enrolling in the intervention Run the logistic propensity score model – With dependent variable: whether the individual is in the treatment group – Include the aforementioned model covariates that might affect the probability of being treated 10
Propensity Score Model Results Analysis of Maximum Likelihood Estimates Standard Parameter Estimate Error P Value Intercept -0.718 0.117 <0.001 Age -0.012 0.002 <0.001 Male -0.056 0.026 0.034 White -0.833 0.045 <0.001 Disabled 0.033 0.042 0.426 ESRD -0.629 0.177 0.000 Number of dual eligible months in the previous calendar year 0.003 0.003 0.439 Number of chronic conditions -0.044 0.005 <0.001 Total payments in calendar quarter prior to enrollment 0.000 0.000 0.002 Total payments in second, third, fourth, and fifth calendar quarters prior to enrollment 0.000 0.000 0.001 Number of ED visits in calendar quarter prior to enrollment 0.044 0.025 0.076 Number of ED visits in second, third, fourth, and fifth calendar quarters prior to enrollment 0.053 0.008 <0.001 Number of inpatient stays in calendar quarter prior to enrollment -0.021 0.054 0.698 Number of inpatient stays in second, third, fourth, and fifth calendar quarters prior to enrollment -0.073 0.026 0.005 11
Propensity Score Matching Estimate propensity scores (PSCORE) using logistic regression. If caliper matching, set the caliper width or maximum propensity score distance (e.g., 20% of the standard deviation of the logit of the propensity scores, Austin, 2011b). 𝑄𝑇𝐷𝑃𝑆𝐹 – 𝑀𝑃𝐻𝐽𝑈_𝑄𝑇𝐷𝑃𝑆𝐹 = log 1−𝑄𝑇𝐷𝑃𝑆𝐹 Within the caliper, we did 1:variable matching (with replacement) with up to three comparison beneficiaries per treatment individual. 12
Propensity Score Matching Results Before Matching After Matching Comparison Treatment Comparison Treatment Group Group Group Group Standardized Standardized Variable Mean SD Mean SD Difference Mean SD Mean SD Difference Total payments in calendar quarter $2,321 $7,363 $1,988 $6,312 0.05 $2,320 $7,363 $2,284 $7,440 0.00 prior to enrollment Total payments in second, third, fourth, and fifth calendar quarters $8,039 $18,720 $7,542 $15,538 0.03 $8,032 $18,713 $8,229 $18,649 0.01 prior to enrollment Age 67.17 15.28 71.04 12.27 0.28 67.18 15.27 67.02 14.65 0.01 Percentage male 42.88 49.49 43.32 49.55 0.01 42.89 49.50 43.13 49.53 0.00 Percentage white 89.30 30.91 95.66 20.38 0.24 89.32 30.89 88.83 31.50 0.02 Percentage disabled 35.00 47.70 26.11 43.92 0.19 34.99 47.70 36.02 48.01 0.02 Percentage ESRD 0.59 7.64 0.68 8.21 0.01 0.59 7.64 0.64 7.96 0.01 Number of dual eligible months in 2.56 4.75 1.90 4.27 0.15 2.55 4.75 2.71 4.88 0.03 the previous calendar year Number of chronic conditions 6.12 3.68 6.85 3.68 0.20 6.12 3.68 6.22 3.69 0.03 Number of ED visits in calendar 0.19 0.72 0.13 0.48 0.10 0.18 0.65 0.17 0.62 0.02 quarter prior to enrollment Number of ED visits in second, third, fourth, and fifth calendar quarters 0.93 2.33 0.65 1.62 0.14 0.92 2.08 0.89 2.21 0.01 prior to enrollment Number of inpatient stays in 0.09 0.38 0.08 0.34 0.04 0.09 0.38 0.09 0.38 0.00 calendar quarter prior to enrollment Number of inpatient stays in second, third, fourth, and fifth calendar 0.29 0.84 0.28 0.76 0.02 0.29 0.84 0.30 0.83 0.00 quarters prior to enrollment — — — — — — Number of beneficiaries 6,478 85,198 6,477 19,431 — — — — — — Number of unique beneficiaries 6,478 85,198 6,477 16,710 — — — — — — — — Number of weighted beneficiaries 6,477 6,477 13
Recommend
More recommend
