Methodological Critiques of the ProPublica Surgeon Scorecard __________________________________ Mark W. Friedberg RAND Corporation, Boston, MA @MWFriedberg #ARM16
Surgeon Scorecard has admirable goals • We need valid and reliable measures of provider performance in general • Choosing your surgeon: a high-stakes decision • Status quo is unsatisfactory – There’s a sense that doctors/nurses/etc. know who the better surgeons are, keep this info from patients • Is this actually true? – Few surgeons track their outcomes • Wouldn’t it be better for patients to be well -informed? 2
Surgeon Scorecard’s problems arise from execution, not intent • Limitations of source data • Methodological decisions – Measure construction – Handling of hospital effects – Case-mix adjustment – Reliability 3
Source data: patient-to-surgeon attribution questionable • Surgeon Scorecard assigned patients to surgeons based on Medicare Part A claims only • Dowd et al (2012) found 28% “operating NPI” mismatch between Part A and Part B claims – Is this random? Unknown. – How many apparent mismatches are due physician vs. group NPIs? Unknown. • Might explain why Surgeon Scorecard included non- surgeons or surgeons in non-applicable specialties • Solution: Assess the validity of attribution 4
Scorecard’s “adjusted complication rates” are not complication rates • New measure, never validated • Composite: 93% readmissions for conditions plausibly related to surgery + 7% deaths – Deaths and readmissions get equal weight • Underestimates absolute rate of complications – Excludes complications during index admission, other than death – Misses ~60-90% of all 30-day complications, depending on procedure • No evidence of association with relative risk of complications, when comparing between surgeons 5
Surgeon Scorecard validation data, so far Source: Auffenberg GB et al, JAMA Surgery, 2016 6
Scorecard’s “adjusted complication rates” are not complication rates • Solution: Rename the measure and validate it 7
Controlling for hospital random effects invalidates between-hospital comparisons • Surgeon Scorecard used hierarchical model with hospital and surgeon-level random effects – But “adjusted complication rates” were generated by setting hospital effects to zero • Patients can’t choose surgeons in hypothetical average hospital • Deeper point: hospital random effects cannot distinguish surgeon recruitment from other hospital characteristics • Solution: Don’t set the hospital random effects to zero 8
Adequacy of case-mix adjustment unclear • Surgeon Scorecard case-mix variables: – Patient age, sex, Elixhauser index-based health score – 1 additional variable for 5 of 8 procedures • Example: use of surgical robot for prostatectomy • Health score coefficient estimate = 0 • Very different from other surgical CMA methods • Solution: Validate the case-mix adjustment, a la Hospital Compare mortality reports 9
No minimum reliability requirement • Reliability unknown, but Laparoscopic cholecystectomy looks low • Misclassification risk unknown, but looks high • Solution: Calculate reliability, set a minimum for reporting Source: Pierce O, Allen M. Assessing surgeon-level risk of patient harm during elective surgery for public reporting: Appendixes to white paper, 2015 10
Concluding thoughts on the Surgeon Scorecard • Is a report with these problems better than no report at all? – People might have different perspectives – But we all should agree on the facts of the case (i.e., the science). Do we? • What can health services research offer? – Identification of validity and reliability limitations – Guidance on how to address these limitations • Hospital effects solution = easy – Help with assessing validity before publication – Post-publication peer review 11
Thank you Contact: Mark Friedberg, MD, MPP mfriedbe@rand.org @MWFriedberg 12
Recommend
More recommend
