Non-linear Difference-in-Differences Models for Policy and Program - - PowerPoint PPT Presentation

Non-linear Difference-in-Differences Models for Policy and Program Evaluation

Claude M. Setodji, Justin W. Timbie, Rosalie Malsberger, Katherine Giuriceo, Katherine L. Kahn

June 25, 2017

Supported by CMS Contract #HHSM-500-2005-00028I Task Order #T0008. The contents of this presentation are solely the responsibility of the authors and do not necessarily represent the official views of the U.S. Department of Health and Human Services or any of its agencies.

2

Outline

 Challenges associated with non-linear difference-in-differences (DID) models  Puhani’s non-linear DID estimator  Simulated example

 Estimating effect of patient-centered medical home recognition on expenditures (modeled using log transformation)

 Recommendations

3

Difference-in-Differences Methods are a Popular Choice for Program Evaluation

 In a DID design, outcomes are assessed both before and after program implementation in both an intervention and control group

 Comparisons are made between trends in both groups with the interaction between group and time being the parameter of interest in the model

 DID is commonly used because of its strengths:

 Mimics random assignment for baseline similarity  Allows estimation of causal inference  Controls for unmeasured confounders making it more powerful than cross-sectional studies

4

Non-linear Models Produce Estimates That Are More Difficult to Interpret

 Non-linear models require transformations to achieve normality

 Log transformations are often used when modeling expenditures, to account for skewed distributions  Estimates of program impacts are on the scale of log dollars, which are difficult to interpret

 “Recycled prediction” or marginal estimates can express estimates on a meaningful scale

5

Recycled Prediction Uses Model Estimates to Set Up Two Counterfactuals

 Two Counterfactual Questions  What if all patients received the intervention?  What if no patients received the intervention?

 Methods  Expenditures are estimated under both scenarios assuming model estimates are correct, after back transformation from log

 The difference is a good proxy for the intervention’s main effect estimate  However, in DID context, the difference representing the interaction can be misleading

6

Naïve Non-Linear DID Estimator

 In the DID model, there are 4 cells

 (1) Baseline, Intervention  (2) Baseline, Control  (3) Follow up, Intervention  (4) Follow up, Control

 A simple approach would generate recycled predictions for each cell, and then estimate:

 Change in Intervention from baseline to follow up  Change in Control from baseline to follow up  DID Estimate = Difference of the two differences

7

Naïve Estimator is Misleading for Non-Linear Models

 The Problem: The log of a difference is not the difference of the logs

 Monotonicity is lost with log transformation  The model coefficients are correct, but the recycled predictions are not

 Proposed solution: The Puhani solution  Puhani uses the interaction as a single variable  Use it like a predictor variable that can be turned on and off for recycling and estimating differences

8

Puhani DID Estimator

 Uses two recycled predictions:

 Interaction “turned on.” Set everyone to have interaction value 1  Interaction “turned off.” Set everyone to have interaction value 0  Take the difference in prediction as a representation of the interaction estimate

 Can be estimated using margins command in Stata

9

Simulated Example of Two DID Estimators

 We use simulated data from the evaluation of CMS’s FQHC Advanced Primary Care Practice Demonstration to show an extreme case, to assess how Patient-Centered Medical Home (PCMH) recognition affects annual expenditures  Simulated data were modeled after Part A and B fee-for-service Medicare claims data

 428,146 adult Medicare beneficiaries with ≥1 visit to a national sample of 1330 FQHC clinics  Beneficiary outcomes and PCMH recognition were simulated from patient and clinic characteristics

 Hypothesis: Recognition will lead to decreased expenditures

10

Beneficiary Characteristics in Simulated Sample

Clinics with PCMH Recognition Clinics without PCMH Recognition Age (in years) 18-64 45.5 44.1 65-74 34.2 34.7 75-84 15.5 16.0  85 4.8 5.2 Race/Ethnicity White 69.1 69.6 Black 17.2 18.4 Hispanic 6.4 7.1 Other/ Unknown 7.2 4.8 Male 54.5 54.0 Disabled 52.3 51.2

11

Skewed Expenditure Needs Transformation

 Expenditure is skewed to the right, requiring

• utcome transformation

 Log transformation of expenditure returned a normal distribution

12

Log Scale Results

 FQHC clinics with PCMH recognition have smaller

increases in expenditures than non-recognized clinics

 Recognition reduces expenditures by -0.227 log dollars

Log scale  P-value PCMH Recognition 0.926 0.000 Post 0.922 0.000 PCMH recognition x Post

• 0.227

0.000 Age 0.005 0.000 Race: Black 0.103 0.012 Hispanic 0.067 0.112 Other/ Unknown 0.170 0.001 Male 0.033 0.183

13

Naïve Recycled Prediction Result

• The interaction coefficient was -0.227 while the “naïve”

recycled estimate is $190.11

• Both are statistically significant; but one is positive one is

negative

Before and After Baseline Follow up Difference

No PCMH Recognition

$186.03 [163.19, 208.81] $467.73 [410.58, 524.92] $281.70 [245.84, 317.61]

PCMH Recognition

$469.79 [411.09, 528.51] $941.60 [823.82, 1059.38] $471.81 [405.77, 537.72]

Difference

$283.76 [244.18, 323.31] $473.87 [402.90, 544.79] + $190.11 [145.78, 234.32]

14

Puhani’s Recycled Prediction Result

 The interaction is effectively modeled as a binary variable the same way we use intervention/control  PCMH recognition is associated with a reduction in expenditures of $106.69

 The interaction coefficient was -0.227 Recycled counterfactual framework Estimate 95% CI Interaction = 0 $525.92 [461.39, 590.41] Interaction = 1 $419.23 [366.80, 471.72] Difference

• $106.69

[-135.35, -78.03]

15

Summary and Recommendations

 Policymakers require easy to interpret estimates of program impacts—especially from non-linear models  Recycled predictions involving interactions can produce misleading conclusions unless appropriate estimators are used  The Puhani estimator can be used with any non-linear monotonic transformation and can be implemented easily in SAS or STATA

16