Validity & EHR-based Clinical Trials Bradley G Hammill Duke - - PowerPoint PPT Presentation

Validity & EHR-based Clinical Trials

Bradley G Hammill Duke School of Medicine & Duke Clinical Research Institute brad.hammill@duke.edu

The Plan

• Introduce ADAPTABLE trial & PCORnet data
• Discuss process-based threats to validity
• Discuss data-based threats to validity

ADAPTABLE trial

• Aspirin Dosing: A Patient-centric Trial Assessing Benefits

and Long-Term Effectiveness – Pragmatic clinical trial – Demonstration project of PCORnet – Patient-level randomization – Leveraging EHR data – Events of interest primarily hospitalization-based – 20+ sites “…designed to reflect ‘real-world’ medical care by recruiting broad populations of patients, embedding the trial into the usual healthcare setting, and leveraging data from health systems to produce results that can be readily used to improve patient care.”

National Patient-Centered Clinical Research Network (PCORnet)

• Distributed Research Network

– 13 Clinical Data Research Networks (CDRNs) comprising 80+ sites – Primarily electronic health record data – Use of Common Data Model (CDM) – Control of data is local, not central – Queries are used to generate summary results for return

PCORnet Common Data Model (CDM)

PCORnet Common Data Model (CDM)

Limitations of EHR Data

• Gaps in data capture exist…

– For certain types of events – For out-of-system encounters

• …that can lead to immediate validity issues

– True event rate – Powered sample size – Site-level confounding Patient X

• Actual

– Apr 2017, Recruited by Duke into study – Jun 2017, hospitalized @ Duke – Aug 2017, hospitalized @ UNC – Jan 2018, dies

• Duke EHR

– Jun 2017, hospitalized @ Duke

Addressing Limitations of EHR Data

• Addressing these gaps

– Data linkage to outside sources – Pre-study gap analysis & selective site recruitment – Loyalty cohorts [observational studies]

Linkages within ADAPTABLE

• For ascertainment of events

– Medicare claims data – Private health plan claims data (selected) – National Death Index – Direct records request

Medicare Claims Data

• Description

– Medicare claims data reflect reimbursement for services requested by providers for beneficiaries enrolled in the traditional (fee-for-service) Medicare program

• Coverage

– Subjects enrolled in fee-for-service Medicare (old age -or- disability)

• Known or anticipated limitations

– Requires known & accurate linking information – Events ascertained using coding algorithms – Quarterly data is ~92% complete

• Final / complete CY data is further delayed

– ~25% of Medicare population is enrolled in a managed care plan (i.e., no claims)

Private Health Plan Claims Data

• Description

– PCORi has funded a demonstration project with Anthem and Humana to provide health plan data for ADAPTABLE subjects

• Coverage

– Subjects enrolled in an Anthem or Humana health plan

• Known or anticipated limitations

– Requires known & accurate linking information – Events ascertained using coding algorithms – Limited geographic coverage – Turnover within plans can be substantial

National Death Index Data

• Description

– The National Death Index is a centralized database of death record information on file in state vital statistics offices

• Coverage

– All subjects

• Known or anticipated limitations

– Requires known & accurate linking information – Early release data is ~90% complete

• Final / complete CY data is further delayed

Direct Records Request

• Description

– Patient-reported events that cannot be reconciled using

• ther sources will be followed up on by the ADAPTABLE

call center

• Coverage

– All subjects

• Known or anticipated limitations

– Requires patient report to trigger reconciliation – Paper records will be returned & events adjudicated

Data Latency

• 𝑢𝐸𝐵𝑈𝐵 : Time delay between latest available data and

acquisition – Present for some sources – Differs by source – Possible reasons: Accrual time; request processing time

• 𝑢𝑄𝑆𝑃𝐷𝐹𝑇𝑇 : Time required for pre-processing data at

coordinating center – Present for some sources (esp. claims) – Assuming uniform for all sources when present (1 month)

• 𝑢𝑇𝑈𝐵𝑈𝑇 : Time required for processing and analyzing data

– Uniform for all sources

𝑢𝑇𝑈𝐵𝑈𝑇 DSMB

Data available

𝑢𝑄𝑆𝑃𝐷𝐹𝑇𝑇 𝑢𝐸𝐵𝑈𝐵

Data Latency

• PCORnet EHR data

– 𝑢𝐸𝐵𝑈𝐵 : Best case ~1 month; worst case ~7+ months

• DataMarts refreshed every 6 months
• Some source tables may not be up-to-date at time of refresh
• Must be certified for use by PCORnet operations center

– 𝑢𝑄𝑆𝑃𝐷𝐹𝑇𝑇 required? No

• Medicare claims data

– 𝑢𝐸𝐵𝑈𝐵 : Best case ~6 months; worst case ~9 months

• Quarterly data available ~5 months following the end of the

quarter

• Acquisition time required (~1 month)

– 𝑢𝑄𝑆𝑃𝐷𝐹𝑇𝑇 required? Yes (~1 month)

𝑢𝑇𝑈𝐵𝑈𝑇 DSMB

Data available

𝑢𝑄𝑆𝑃𝐷𝐹𝑇𝑇 𝑢𝐸𝐵𝑈𝐵

Data Latency

• Private health plan claim data

– 𝑢𝐸𝐵𝑈𝐵 ~4 months

• ~3-month lag in claims to insurer
• Acquisition time required (~1 month)

– 𝑢𝑄𝑆𝑃𝐷𝐹𝑇𝑇 required? No

• National Death Index data

– 𝑢𝐸𝐵𝑈𝐵 : Best case ~2 months; worst case ~13+ months

• Early release data available 2-3 months after the end of the

CY

• Acquisition time required (~1 month)

– 𝑢𝑄𝑆𝑃𝐷𝐹𝑇𝑇 required? Yes (~1 month)

• Assuming 𝑢𝑇𝑈𝐵𝑈𝑇 = 1 month for all data sources

𝑢𝑇𝑈𝐵𝑈𝑇 DSMB

Data available

𝑢𝑄𝑆𝑃𝐷𝐹𝑇𝑇 𝑢𝐸𝐵𝑈𝐵

Impact of Data Latency

Data Source Data Through… Mytrus Patient Portal 1-Nov-2017 PCORnet DataMart (recent refresh) 1-Oct-2017 PCORnet DataMart (distant refresh) 1-Apr-2017 Medicare Claims Data 30-Jun-2017 National Death Index 31-Dec-2016 Private Health Plan Data 1-Aug-2016

Dec-2017 DSMB Apr-2016

Information Asynchrony

• Events found in EHR / Medicare / PHP data are accepted as true
• Patient-reported events must be reconciled

– Search EHR? Found = Confirmed. Unfound… – Search Medicare? Found = Confirmed. Unfound… – Search Private Health Plan data? Found = Confirmed. Unfound… – Call for medical records

• Data latency affects timing of event recording (by trial) and reconciliation

Information Asymmetry

• By type of data

– Raw hospital records vs. coded hospital records

• By sources of data

– Different patients can have different sources of data contributing to endpoint ascertainment

EHR CMS NDI HP Patient #1 EHR CMS NDI HP Patient #2 EHR CMS NDI HP Patient #3 EHR CMS NDI HP Patient #4

Potential Data Issues in PCORnet

• Validity of coded endpoints
• Quality of data at PCORnet sites
• Identifying appropriate patients for recruitment (computable phenotype)

Validity of Coded Endpoints

• ADAPTABLE events

– Death – Hospitalization for non-fatal MI – Hospitalization for stroke – Coronary revascularization – Hospitalization for major bleeding

• Not exclusively an EHR issue, but…
• Definitions vastly different from “regular” trials

Definitions of Myocardial Infarction

EHR criteria Adjudication criteria

Inpatient encounter w/ ICD-9-CM diagnosis code 410.x0, 410.x1 in primary position ECG or changes consistent with acute infarction or ischemia MI:

• New diagnostic Q waves (Q wave in leads

V2 and V3 ≥ 0.02 sec or QS complex in leads V2 and V3; Q wave ≥ 0.03 sec and ≥ 0.1 mV deep or QS complex in leads I, II, aVL, aVF

• r

V4-V6 in any two leads of a contiguous lead grouping (I and aVL; V1-V6; II, III, aVF, R wave ≥ 0.04 sec in V1 and V2 and R/S ≥ 1 with a concordant positive T wave)) in the absence of conduction abnormalities

• New significant ST
• segment-T
• wave changes in two or more contiguous leads: ST

elevation at the J point ≥ 0.1 mV in all leads other than leads V2 and V3 where the following cut points apply: ≥ 0.2 mV in men ≥ 40 years; 0.25 mV in men < 40 years, or ≥ 0.15 mV in women. ST depression horizontal or downsloping ≥ 0.05 mV; or T wave inversion ≥ 0,1mV with prominent R wave or R/S ratio ≥ 1.

• Development of new left bundle branch block (LBBB)
• Imaging evidence of new loss of viable myocardium or new regional wall motion

abnormality

• Intracoronary thrombus by angiography

AND Elevated cardiac biomarkers (values according to each hospital’s laboratory): A rise and/or fall in cardiac biomarker values (preferably troponin, CKMB, AST, LDH or myoglobin) with at least one value above the 99th percentile of the upper reference limit.

ADAPTABLE Validation Studies

• EHR-coded events vs. adjudicated events
• Patient-reported events vs. coded events

Related PCORnet Hospitalization Issue

• Not all sites code “primary” diagnosis for hospitalizations

– Effect = No events at a site?

• Address by…

– Making this a required field for site inclusion – Defining alternative endpoints

• Myocardial infarction

– Primary: Inpatient encounter w/ICD-9-CM diagnosis code 410.x0, 410.x1 in primary position – Alternate: Inpatient encounter w/ICD-9-CM diagnosis code 410.x0, 410.x1 in any position

Quality of Data at PCORnet Sites

• Examples:

– Mistakes in data – Missing specific types of data or encounters

• PCORnet Data Characterization process

– Gross data checks to examine quality

Identifying Appropriate Patients for Recruitment

• Computable phenotype

– Identify patients eligible for trial using EHR data – Only as good as the source data – May require direct patient inquiry – Different levels of certainty in coded definitions

ADAPTABLE Inclusion/Exclusion Criteria

• Known atherosclerotic cardiovascular disease (ASCVD), defined by a

history of prior myocardial infarction, prior coronary angiography showing ≥75% stenosis of at least one epicardial coronary vessel, or prior coronary revascularization procedures (either PCI or CABG)

• Age ≥ 18 years
• No known safety concerns or side effects considered to be related to

aspirin, including – No history of significant allergy to aspirin such as anaphylaxis, urticaria, or significant gastrointestinal intolerances – No history of significant GI bleed within the past 12 months – Significant bleeding disorders that preclude the use of aspirin

• Access to the Internet.
• Not currently treated with an oral anticoagulant and not planned to be

treated in the future with an oral anticoagulant for existing indications such as atrial fibrillation, deep venous thrombosis, or pulmonary embolism.

• Not currently treated with ticagrelor and not planned to be treated in

the future with ticagrelor.

• Female patients who are not pregnant or nursing an infant
• Estimated risk of a major cardiovascular event (MACE) > 8% over next 3

years as defined by the presence of at least one or more of the following enrichment factors: – Age > 65 years – Serum creatinine > 1.5 mg/dL – Diabetes mellitus (Type 1 or Type 2) – 3-vessel coronary artery disease – Cerebrovascular disease and/or peripheral arterial disease – Left ventricular ejection fraction (LVEF) < 50% – Current cigarette smoker

ADAPTABLE Inclusion/Exclusion Criteria

• Known atherosclerotic cardiovascular disease (ASCVD), defined by a

history of prior myocardial infarction, prior coronary angiography showing ≥75% stenosis of at least one epicardial coronary vessel, or prior coronary revascularization procedures (either PCI or CABG)

• Age ≥ 18 years
• No known safety concerns or side effects considered to be related to

aspirin, including – No history of significant allergy to aspirin such as anaphylaxis, urticaria, or significant gastrointestinal intolerances – No history of significant GI bleed within the past 12 months – Significant bleeding disorders that preclude the use of aspirin

• Access to the Internet.
• Not currently treated with an oral anticoagulant and not planned to be

treated in the future with an oral anticoagulant for existing indications such as atrial fibrillation, deep venous thrombosis, or pulmonary embolism.

• Not currently treated with ticagrelor and not planned to be treated in

the future with ticagrelor.

• Female patients who are not pregnant or nursing an infant
• Estimated risk of a major cardiovascular event (MACE) > 8% over next 3

years as defined by the presence of at least one or more of the following enrichment factors: – Age > 65 years – Serum creatinine > 1.5 mg/dL – Diabetes mellitus (Type 1 or Type 2) – 3-vessel coronary artery disease – Cerebrovascular disease and/or peripheral arterial disease – Left ventricular ejection fraction (LVEF) < 50% – Current cigarette smoker

Summary

• Threats to validity lurk around all corners of an EHR-based

clinical trial – Process-based – Data-based

• Understanding these threats is the first step to addressing them
• We’re still learning the best way to handle

Thank you! Questions?