External Validity of NYC Macroscope Electronic Health External - - PowerPoint PPT Presentation

External Validity of NYC Macroscope Electronic Health External Validity of NYC Macroscope Electronic Health External Validity of NYC Macroscope Electronic Health External Validity of NYC Macroscope Electronic Health Record Surveillance System Indicator Definitions of Obesity, Record Surveillance System Indicator Definitions of Obesity, Record Surveillance System Indicator Definitions of Obesity, Record Surveillance System Indicator Definitions of Obesity, Smoking, Diabetes, Hypertension and Hypercholesterolemia Smoking, Diabetes, Hypertension and Hypercholesterolemia Smoking, Diabetes, Hypertension and Hypercholesterolemia Smoking, Diabetes, Hypertension and Hypercholesterolemia

Katharine H. McVeigh, PhD, MPH New York City Department of Health and Mental Hygiene Presented at Concordium September 13, 2016

NYC Macroscope Team NYC Macroscope Team NYC Macroscope Team NYC Macroscope Team

NYC Department of Health and Mental Hygiene Katharine H. McVeigh, PhD, MPH Sharon E. Perlman, MPH Elizabeth Lurie, MPH Pui Ying Chan, MPH Kathleen Tatem, MPH Sungwoo Lim, DrPH Laura Jacobson, MSPH Lauren Schreibstein, MA City University of New York School of Public Health Lorna E. Thorpe, PhD

Special thanks to Jay Bala, Katherine Bartley, Claudia Chernov, Amy Freeman, Ryan Grattan, Carolyn Greene, Tiffany Harris, Stephen Immerwahr, Kevin Konty, Ram Koppaka, Remle Newton-Dame, Jesica Rodriguez-Lopez, Matthew Romo, Sarah Shih, Jesse Singer, Elisabeth Snell

This work has been made possible by the financial support of the de Beaumont Foundation, the Robert Wood Johnson Foundation and its National Coordinating Center for Public Health Services and Systems Research, the Robin Hood Foundation, the NY State Health Foundation, the Doris Duke Charitable Foundation, and the Centers for Disease Control and Prevention (U28EH000939)

Electronic Health Records (EHRs) Can Transform Electronic Health Records (EHRs) Can Transform Electronic Health Records (EHRs) Can Transform Electronic Health Records (EHRs) Can Transform Medical Records into Actionable Information Medical Records into Actionable Information Medical Records into Actionable Information Medical Records into Actionable Information

20 30 40 50 60 70 80 Oct-09 Dec-09 Feb-10 Apr-10 Jun-10 Aug-10 Oct-10 Dec-10 Feb-11 Apr-11 Jun-11 Aug-11 Oct-11

Why is Electronic Health Record Why is Electronic Health Record Why is Electronic Health Record Why is Electronic Health Record-

• Based

Based Based Based Surveillance Important? Surveillance Important? Surveillance Important? Surveillance Important?

• The burden of chronic disease in the U.S. is

increasing.

• Population-level monitoring of disease and risk factor

prevalence is important for prevention and mitigation

• Traditional surveys are becoming more difficult to

carry out

• Telephone survey response rates are dropping
• Examination surveys are extremely expensive, labor

intensive, and often have lengthy lag times between data collection and dissemination

What is the NYC Macroscope? What is the NYC Macroscope? What is the NYC Macroscope? What is the NYC Macroscope?

• An electronic health record-based surveillance

system for New York City, focusing on chronic disease and risk factors

• Developed by the New York City Department of

Health and Mental Hygiene in collaboration with the City University of New York School of Public Health

Key Features of NYC Macroscope (1) Key Features of NYC Macroscope (1) Key Features of NYC Macroscope (1) Key Features of NYC Macroscope (1)

• Based on a distributed data model
• Hub Population Health System
• eClinicalWorks EHR platform
• Inclusion/exclusion criteria
• Practice – Documentation quality thresholds
• Provider – Primary care only
• Patient – Visit in 2013, ages 20-100, sex recorded as

male or female, NYC Zip Code

• Record – Lab measures require electronic reporting,

records with missing data are dropped before weighting

Key Features of NYC Macroscope (2) Key Features of NYC Macroscope (2) Key Features of NYC Macroscope (2) Key Features of NYC Macroscope (2)

• Weighted to the distribution of the NYC adult

population that had seen a health provider in the past year

• N = 716,076 patients seen in 2013 (17.5% of 4.1 M in care)
• Age group (20-39, 40-59, 60-100)
• Sex (male, female)
• Neighborhood poverty level (< 10%, 10-19%, 20-29%, >= 30%)
• Validated against 2 population-based reference surveys
• 2013-14 NYC Health and Nutrition Examination Survey (NYC

HANES)

• N = 1,524; 1,135 in care
• 2013 NYC Community Health Survey (CHS)
• N* = 8,356; 6,166 in care

Population Estimate Comparisons Population Estimate Comparisons Population Estimate Comparisons Population Estimate Comparisons

10 20 30 40 50 60 Obesity Smoking Diabetes Diagnosis *Augmented Diabetes Hypertension Diagnosis *Augmenented Hypertension Hypercholesterolemia Diagnosis *Augmented Hypercholesterolemia

NYC Macroscope, NYC HANES and CHS Prevalence Estimates (%)

NYC Macroscope NYC HANES CHS * Not available in CHS

Medical Chart Review Medical Chart Review Medical Chart Review Medical Chart Review Criterion Criterion Criterion Criterion-

• Related

Related Related Related Validity (N = 48) Validity (N = 48) Validity (N = 48) Validity (N = 48)

Sensitivity

0.2 0.4 0.6 0.8 1 Augmented Hypercholesterolemia Hypercholesterolemia Diagnosis Augmenented Hypertension Hypertension Diagnosis Augmented Diabetes Diabetes Diagnosis Smoking Obesity

Specificity

0.2 0.4 0.6 0.8 1 Augmented Hypercholesterolemia Hypercholesterolemia Diagnosis Augmenented Hypertension Hypertension Diagnosis Augmented Diabetes Diabetes Diagnosis Smoking Obesity

External Validity Study External Validity Study External Validity Study External Validity Study

Is the validity of NYC Macroscope indicators generalizable to EHR data maintained by other providers on other platforms?

Research Questions Research Questions Research Questions Research Questions

• What is the criterion-related validity of NYC

Macroscope indicator definitions in data from practices that do not contribute to the NYC Macroscope?

• Does validity improve if records are restricted to

providers who have attested to stage 1 meaningful use?

Participant Inclusion/Exclusion Flow Participant Inclusion/Exclusion Flow Participant Inclusion/Exclusion Flow Participant Inclusion/Exclusion Flow Chart Chart Chart Chart

Enrolled in NYC HANES 2013-14 N=1,524 Had a doctor visit in past year n=1,135 Signed consent n=692 Signed HIPAA waiver n=491 One or more EHRs obtained n=277 EHR contained valid data n=190

Not in care n=389 No consent n=443 No HIPAA waiver n=201 No EHR, no visits, specialist, unable to locate, not released n=214 Excluded provider type n=87 NYC Macroscope records n=48 Non-Macroscope records n=142

Methods Methods Methods Methods

• 142 non-Macroscope records
• 133 providers
• 89 medical practices
• > 20 different EHR vendor platforms
• Sensitivity and Specificity
• Full sample
• Sub-sample of 86 records (MU1)

Sample Sample Sample Sample Characteristics Characteristics Characteristics Characteristics

Non-Macroscope Records NYC Macroscope (n=48) % All Records (n=142) % MU1 Subsample (n=86) %

Age Group 20-39 40-49 ≥ 60 35 46 19 36 37 28 35 34 31 Sex Female Male 65 35 65 35 70 30 Neighborhood Poverty < 10% 10%-19% 20%-29% ≥ 30% 27 33 21 19 23 36 25 16 21 34 31 14

Sensitivity Sensitivity Sensitivity Sensitivity

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Augmented Hypercholesterolemia Hypercholesterolemia Diagnosis Augmenented Hypertension Hypertension Diagnosis Augmented Diabetes Diabetes Diagnosis Smoking Obesity All Non-Macroscope Records (n=142) Non-Macroscope with MU1 Restriction (n=86) NYC Macroscope (n=48)

Validity threshold ≥ 0.70

Specificity Specificity Specificity Specificity

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Augmented Hypercholesterolemia Hypercholesterolemia Diagnosis Augmenented Hypertension Hypertension Diagnosis Augmented Diabetes Diabetes Diagnosis Smoking Obesity All Non-Macroscope Records (n=142) Non-Macroscope with MU1 Restriction (n=86) NYC Macroscope (n=48)

Validity threshold ≥ 0.80

Summary Summary Summary Summary

• Both indicators of hypercholesterolemia performed

poorly

• All other measures performed well
• The above conclusions are consistent with findings

from the 48 NYC Macroscope records

• Restricting records to those from providers who have

attested to stage 1 meaningful use meaningfully improved the sensitivity of the smoking and hypertension diagnosis indicators

Strengths and Limitations Strengths and Limitations Strengths and Limitations Strengths and Limitations

• Strengths
• Heterogeneity of providers (N = 133) and EHR vendor

platforms (N > 20)

• Innovative sample and gold standard criterion
• Limitations
• Small sample size/large confidence intervals

Conclusion Conclusion Conclusion Conclusion

• NYC Macroscope indicator definitions measuring outcomes other

than hypercholesterolemia demonstrated good external validity in a sample of 142 records from 133 providers and recorded on more than 22 different EHR platforms.

• NYC Macroscope indicator definitions of obesity and diabetes

demonstrated consistently high external validity and can be adopted by other jurisdictions with minimal local validation.

• The validity of NYC Macroscope indicator definitions of smoking

and hypertension varied across samples from acceptable to high. Sensitivity may be higher if documentation quality criteria such as those required by Meaningful Use are incorporated into the indicator definitions.

Next Steps Next Steps Next Steps Next Steps

• Analysis of NYC Macroscope trends over time
• Incorporation of race stratification into weighting and

analysis

• Development of an early childhood obesity module
• Comparison of direct and modeled estimates of

neighborhood prevalence

• Investigation of reasons for poor validity of cholesterol

indicators

Thank You! Thank You! Thank You! Thank You!

For more information, Google “NYC Macroscope” or contact us at: NYCMacroscope@health.nyc.gov