__________________________________ Mark W. Friedberg RAND - - PowerPoint PPT Presentation

Methodological Critiques of the ProPublica Surgeon Scorecard

__________________________________

Mark W. Friedberg RAND Corporation, Boston, MA @MWFriedberg #ARM16

• We need valid and reliable measures of provider

performance in general

• Choosing your surgeon: a high-stakes decision
• Status quo is unsatisfactory

– There’s a sense that doctors/nurses/etc. know who the better surgeons are, keep this info from patients

• Is this actually true?

– Few surgeons track their outcomes

• Wouldn’t it be better for patients to be well-informed?

2

Surgeon Scorecard has admirable goals

• Limitations of source data
• Methodological decisions

– Measure construction – Handling of hospital effects – Case-mix adjustment – Reliability

3

Surgeon Scorecard’s problems arise from execution, not intent

• Surgeon Scorecard assigned patients to surgeons based
• n Medicare Part A claims only
• Dowd et al (2012) found 28% “operating NPI” mismatch

between Part A and Part B claims – Is this random? Unknown. – How many apparent mismatches are due physician

• vs. group NPIs? Unknown.
• Might explain why Surgeon Scorecard included non-

surgeons or surgeons in non-applicable specialties

• Solution: Assess the validity of attribution

4

Source data: patient-to-surgeon attribution questionable

• New measure, never validated
• Composite: 93% readmissions for conditions plausibly

related to surgery + 7% deaths – Deaths and readmissions get equal weight

• Underestimates absolute rate of complications

– Excludes complications during index admission,

• ther than death

– Misses ~60-90% of all 30-day complications, depending on procedure

• No evidence of association with relative risk of

complications, when comparing between surgeons

5

Scorecard’s “adjusted complication rates” are not complication rates

6

Surgeon Scorecard validation data, so far

Source: Auffenberg GB et al, JAMA Surgery, 2016

• Solution: Rename the measure and validate it

7

Scorecard’s “adjusted complication rates” are not complication rates

• Surgeon Scorecard used hierarchical model with

hospital and surgeon-level random effects – But “adjusted complication rates” were generated by setting hospital effects to zero

• Patients can’t choose surgeons in hypothetical average

hospital

• Deeper point: hospital random effects cannot distinguish

surgeon recruitment from other hospital characteristics

• Solution: Don’t set the hospital random effects to zero

8

Controlling for hospital random effects invalidates between-hospital comparisons

• Surgeon Scorecard case-mix variables:

– Patient age, sex, Elixhauser index-based health score – 1 additional variable for 5 of 8 procedures

• Example: use of surgical robot for prostatectomy
• Health score coefficient estimate = 0
• Very different from other surgical CMA methods
• Solution: Validate the case-mix adjustment, a la Hospital

Compare mortality reports

9

Adequacy of case-mix adjustment unclear

• Reliability unknown, but

looks low

• Misclassification risk

unknown, but looks high

• Solution: Calculate

reliability, set a minimum for reporting

10

No minimum reliability requirement

Source: Pierce O, Allen M. Assessing surgeon-level risk

• f patient harm during elective surgery for public

reporting: Appendixes to white paper, 2015

Laparoscopic cholecystectomy

• Is a report with these problems better than no report at

all? – People might have different perspectives – But we all should agree on the facts of the case (i.e., the science). Do we?

• What can health services research offer?

– Identification of validity and reliability limitations – Guidance on how to address these limitations

• Hospital effects solution = easy

– Help with assessing validity before publication – Post-publication peer review

11

Concluding thoughts on the Surgeon Scorecard

Thank you

12

Contact: Mark Friedberg, MD, MPP mfriedbe@rand.org @MWFriedberg