Prospective, Active Surveillance: The DELTA System User Training - PowerPoint PPT Presentation

Prospective, Active Surveillance: The DELTA System User Training Session October 22, 2018 Frederic S. Resnic, MD MSc Henry Ssemaganda, MD MSc Susan Robbins, BS Comparative Effectiveness Research Institute Lahey Clinic Medical Center 1

Agenda • Introduction to Automated Surveillance (Prospective, Active Surveillance) • Overview of DELTA System • CathPCI Pilot Example (Break) • DELTA User Support • DELTA Demo Interactive Session 2

Introduction to DELTA Automated Safety Surveillance Frederic S. Resnic, MD MSc 3

Challenges of Existing Post-Market Medical Device Surveillance Approaches 1. Adverse event reporting unable to estimate event rates (‘denominator’ issue), non-representative (reporting behavior), resource intensive. 2. Post-Approval Studies generally underpowered, without contemporary comparator groups, for low frequency safety events. 3. Current approaches lack systematic evaluation of real-world Complementary role for performance information to provide Prospective, Near Real-Time timely information to providers, Active Safety Surveillance manufacturers and regulators.

Defining Active Surveillance • The term “ Active Surveillance ” implies differing monitoring strategies to different stakeholders. • CDRH : Non-research efforts that are part of routine post-market vigilance programs for specific (typically high-risk) medical devices. • Industry : Exploratory, non-research, post-market, device performance monitoring. • Academic : Prospective, pre-specified, risk-adjusted, time-limited analysis of RWE. • Key attributes of medical device Active Surveillance • Timely : Primary goal is to assess device performance in time frame that permits appropriate action (by manufacturer, clinicians, and regulators). • Repeated Assessments : Minimize delay in identifying safety signal, or absence of signal. • Methodologically Appropriate : Appropriate risk adjustment, comparators and signal thresholds. Minimize Type II error (false alarms) without missing opportunities to detect real issues.

Defining “ Automated” Surveillance • Automated Surveillance extends Active Surveillance by leveraging software tools to support the pre-specified, repeated evaluations of an accruing dataset. • Automated Surveillance enables: • Near real-time, prospective, evaluation of accruing experience • Simultaneous monitoring of multiple devices and outcomes • Application of robust data security protocols for remote data monitoring. Active Automated Real Time, Retrospective, Surveillance Surveillance Routine, Single Study Continuous Device Device Performance Performance Evaluation Accruing Data, Accruing Data, Monitoring Retrospective, Accruing Data , Near Real-Time Near Real-Time Sequential Sequential Sequential Sequential, Study Study Study Programmed Inexorable progression toward real-time, high quality RWE availability as EHR, Registry/CRN and Informatics technologies mature.

Device Active Surveillance: 10 Years of Learning Lessons JAMIA 2006 JAMA 2010 Circ Card Qual 2012 7 NEJM 2017

CathPCI DELTA Study: Background • Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures. • VCD use variety of mechanical/pharmacologic designs and delivery mechanisms to secure device and accelerate clotting. • PMA studies generally small and limited high quality post-market safety data regarding VCD. • Mynx VCD suspected of having higher failure rates than alternative VCD. • Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014) . • Post-publication analysis of national registry study (Tavris et al - J Invasive Card 2012) .

CathPCI DELTA Study: Background

CathPCI DELTA Study: Hypothesis 1. Active, prospective surveillance of national cardiovascular registry will demonstrate: Mynx VCD is associated with increased rates of post-procedure vascular complications and/or bleeding compared with propensity matched patients receiving alternative VCD. 2. Additionally, Mynx safety signal will be heightened in high risk patient subgroups: women, age>70yrs, and diabetic patients.

CathPCI DELTA Study: Methods Overview • Prospective , observational study using Propensity Matching (1:1) to compare similar populations of patients receiving Mynx following PCI to patients receiving other “active” VCD. • Primary endpoint : Any vascular complication – composite of access site hematoma requiring treatment, access site bleeding requiring intervention or RPH. • Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews. [Avoid “Data mining”] • Pre- specified high risk patient subsets (DM, female, age≥70yr), sensitivity analyses and “falsification hypothesis” analysis. • Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

CathPCI DELTA Study : Patient Flow Diagram 13 Source: Resnic FS, Majithia A, et al. NEJM Jan. 2017

Methods Review: What is Propensity Matching? ALL PCI Patients Receiving Other VCD ALL PCI Patients Receiving Mynx PS Matched Mynx Patients Unmatched Mynx Patients Matched Non-Mynx Patients =

CathPCI DELTA Study: Primary Analysis Mynx VCD was associated with a 59% increased risk of vascular complications as compared with alternative VCD. Signal apparent within 9 months. 15 Source: Resnic FS, Majithia A, et al. NEJM Jan. 2017

CathPCI DELTA Study Results : High Risk Populations Pre-specified subgroup analysis was performed for high risk subgroups including: women, the age>70 and patients with diabetes. Female Patients Age > 70yr Diabetic Patients Source: Resnic FS, Majithia A, et al. NEJM 2017

CathPCI DELTA : Falsification Hypothesis 17 Source: Resnic FS et al. NEJM Jan. 2017 (Supplementary Appendix)

Methodologic Challenge: Selecting Comparators • Multiple options, each with inherent limitations: • Risk-adjusted performance compared with OPC – questions of validity of OPC • Risk-adjusted performance compared with historical Sensitivity Validity performance of alternative therapeutic option - issues around learning impacts and secular treatment trends. • Direct device-device (or product group) comparisons (propensity matched or adjusted) – significant stakeholder sensitivity. • Guiding Principle: Select comparator to maximize validity within limitations of stakeholder sensitivity/tolerance.

Methodologic Challenge: Selecting Methods • Multiple options for analyzing serial outcomes data using surveillance framework: • Propensity matched analysis – robust, addresses confounding, most interpretable. Weaknesses : truly novel technology, multiple comparisons, Generalizability Interpretability treatment selection biases, sample loss. • Propensity adjusted analysis – robust, addresses confounding, maximizes sample. Weaknesses : interpretability, multiple comparisons. • Sequential Testing (CSUM, SPRT, etc) – indefinite surveillance, address multiple comparison, generalizable. Weaknesses : interpretability, risk adjustment • Guiding Principle: Maximize interpretability while minimizing risk of confounding.

Methodologic Challenge: Alert Thresholds • Selecting appropriate thresholds for triggering additional exploration/action is critical. Must be part of written protocol, endorsed by all prior to start of data analysis. • Recommended: Select clinically meaningful difference in performance. • Can differ based on outcome studied: Small differences in mortality with larger differences tolerated with less serious outcomes. • Select alpha (false-positive rate) based on seriousness of outcome. Acceptable to be different from 0.05. • Example : ICD Lead failure: absolute difference ≥ 10% than comparator lead at 1- year, α=0.05. Death alerts at ≥ 5% absolute difference , α= 0.10.

Advantages of DELTA Surveillance 1. Automated Continuous, Prospective, Active Surveillance: suitable for monitoring of large clinical datasets using straightforward analytic methods . 2. S upports multiple, simultaneous, active surveillance analyses . 3. Pre-specified analytic plan: DELTA configured at outset of analysis to perform studies in accordance with SAP/protocol. 4. Propensity matching is uniquely suited to comparative safety analyses. Statistically robust, very easy to interpret and explain, and is conducive to post-hoc analysis for signal exploration . 5. Pragmatic and Scalable Approach. Validated in central data model and distributed models.

Overview of DELTA Frederic S. Resnic, MD MSc and Susan Robbins 25

Overview of DELTA System DELTA Open-Source Version 3.61 • DELTA user interface, with integrated DELTAlytics analytic engine, utilizing MySQL as back-end data repository • Installation packages available on BitBucket 26

Overview of DELTA System JUST TO MENTION…Site Admin Setup • Advanced Features • DELTAlytics statistical package interface setup – part of Installation • Site / Organizations • Users / Permissions/Roles (all Demo Users are Organization Admins) • Projects / Groups • Source Data Connections and Table Relationships (Rely on your data managers) • JDBC databases allowed – MSSQL, MySQL • Requires Database source file – even a Flat File must be imported to db • Alerts and Notifications Note: Topic detail not covered today