Prospective, Active Surveillance: The DELTA System User Training - - PowerPoint PPT Presentation

Frederic S. Resnic, MD MSc Henry Ssemaganda, MD MSc Susan Robbins, BS

Comparative Effectiveness Research Institute Lahey Clinic Medical Center

Prospective, Active Surveillance: The DELTA System User Training Session

October 22, 2018

1

2

• Introduction to Automated Surveillance

(Prospective, Active Surveillance)

• Overview of DELTA System
• CathPCI Pilot Example

(Break)

• DELTA User Support
• DELTA Demo Interactive Session

Agenda

3

Introduction to DELTA Automated Safety Surveillance

Frederic S. Resnic, MD MSc

Challenges of Existing Post-Market Medical Device Surveillance Approaches

• 1. Adverse event reporting unable to

estimate event rates (‘denominator’ issue), non-representative (reporting behavior), resource intensive.

• 2. Post-Approval Studies generally

underpowered, without contemporary comparator groups, for low frequency safety events.

• 3. Current approaches lack systematic

evaluation of real-world performance information to provide timely information to providers, manufacturers and regulators.

Complementary role for Prospective, Near Real-Time Active Safety Surveillance

• The term “Active Surveillance” implies differing monitoring

strategies to different stakeholders.

• CDRH: Non-research efforts that are part of routine post-market

vigilance programs for specific (typically high-risk) medical devices.

• Industry: Exploratory, non-research, post-market, device performance

monitoring.

• Academic: Prospective, pre-specified, risk-adjusted, time-limited

analysis of RWE.

• Key attributes of medical device Active Surveillance
• Timely: Primary goal is to assess device performance in time frame that

permits appropriate action (by manufacturer, clinicians, and regulators).

• Repeated Assessments: Minimize delay in identifying safety signal, or

absence of signal.

• Methodologically Appropriate: Appropriate risk adjustment, comparators

and signal thresholds. Minimize Type II error (false alarms) without missing

• pportunities to detect real issues.

Defining Active Surveillance

• Automated Surveillance extends Active Surveillance by leveraging

software tools to support the pre-specified, repeated evaluations of an accruing dataset.

• Automated Surveillance enables:
• Near real-time, prospective, evaluation of accruing experience
• Simultaneous monitoring of multiple devices and outcomes
• Application of robust data security protocols for remote data monitoring.

Defining “Automated” Surveillance

Retrospective, Single Study Device Performance Evaluation Real Time, Routine, Continuous Device Performance Monitoring Retrospective, Sequential Study Accruing Data, Sequential Study Accruing Data, Near Real-Time Sequential Study Accruing Data, Near Real-Time Sequential, Programmed

Active Surveillance Automated Surveillance Inexorable progression toward real-time, high quality RWE availability as EHR, Registry/CRN and Informatics technologies mature.

JAMIA 2006 JAMA 2010 Circ Card Qual 2012

7

Device Active Surveillance: 10 Years of Learning Lessons

NEJM 2017

CathPCI DELTA Study: Background

• Vascular Closure Devices (VCD) approved for use

following cardiac catheterization procedures.

• VCD use variety of mechanical/pharmacologic designs

and delivery mechanisms to secure device and accelerate clotting.

• PMA studies generally small and limited high quality

post-market safety data regarding VCD.

• Mynx VCD suspected of having higher failure rates

than alternative VCD.

• Observations from 5-hospital prospective safety

surveillance (Kumar et al - Circ Card Qual Outcomes 2014).

• Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012).

CathPCI DELTA Study: Background

CathPCI DELTA Study: Hypothesis

• 1. Active, prospective surveillance of national

cardiovascular registry will demonstrate: Mynx VCD is associated with increased rates of post-procedure vascular complications and/or bleeding compared with propensity matched patients receiving alternative VCD.

• 2. Additionally, Mynx safety signal will be heightened

in high risk patient subgroups: women, age>70yrs, and diabetic patients.

CathPCI DELTA Study: Methods Overview

• Prospective, observational study using Propensity Matching

(1:1) to compare similar populations of patients receiving Mynx following PCI to patients receiving other “active” VCD.

• Primary endpoint: Any vascular complication – composite of

access site hematoma requiring treatment, access site bleeding requiring intervention or RPH.

• Pre-specified protocol and statistical plan including adjustment

for multiple comparisons and planned interim data reviews. [Avoid “Data mining”]

• Pre-specified high risk patient subsets (DM, female, age≥70yr),

sensitivity analyses and “falsification hypothesis” analysis.

• Study oversight committee recommended post-hoc analyses

including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study: Patient Flow Diagram

Source: Resnic FS, Majithia A, et al. NEJM Jan. 2017

Methods Review: What is Propensity Matching?

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD Matched Mynx Patients Unmatched Mynx Patients Matched Non-Mynx Patients PS

=

15

CathPCI DELTA Study: Primary Analysis

Mynx VCD was associated with a 59% increased risk of vascular complications as compared with alternative VCD. Signal apparent within 9 months.

Source: Resnic FS, Majithia A, et al. NEJM Jan. 2017

Pre-specified subgroup analysis was performed for high risk subgroups including: women, the age>70 and patients with diabetes.

Female Patients Age > 70yr Diabetic Patients

CathPCI DELTA Study Results : High Risk Populations

Source: Resnic FS, Majithia A, et al. NEJM 2017

17

CathPCI DELTA: Falsification Hypothesis

Source: Resnic FS et al. NEJM Jan. 2017 (Supplementary Appendix)

• Multiple options, each with inherent limitations:
• Risk-adjusted performance compared with OPC –

questions of validity of OPC

• Risk-adjusted performance compared with historical

performance of alternative therapeutic option - issues around learning impacts and secular treatment trends.

• Direct device-device (or product group) comparisons

(propensity matched or adjusted) – significant stakeholder sensitivity.

• Guiding Principle: Select comparator to maximize validity

within limitations of stakeholder sensitivity/tolerance.

Methodologic Challenge: Selecting Comparators

Validity Sensitivity

Methodologic Challenge: Selecting Methods

• Multiple options for analyzing serial outcomes data using

surveillance framework:

• Propensity matched analysis – robust, addresses

confounding, most interpretable. Weaknesses: truly novel technology, multiple comparisons, treatment selection biases, sample loss.

• Propensity adjusted analysis – robust, addresses

confounding, maximizes sample. Weaknesses: interpretability, multiple comparisons.

• Sequential Testing (CSUM, SPRT, etc) – indefinite

surveillance, address multiple comparison, generalizable. Weaknesses: interpretability, risk adjustment

• Guiding Principle: Maximize interpretability while minimizing

risk of confounding.

Generalizability Interpretability

• Selecting appropriate thresholds for triggering additional

exploration/action is critical. Must be part of written protocol, endorsed by all prior to start of data analysis.

• Recommended: Select clinically meaningful difference in

performance.

• Can differ based on outcome studied: Small differences

in mortality with larger differences tolerated with less serious outcomes.

• Select alpha (false-positive rate) based on seriousness of
• utcome. Acceptable to be different from 0.05.
• Example: ICD Lead failure: absolute difference ≥ 10% than

comparator lead at 1-year, α=0.05. Death alerts at ≥ 5% absolute difference, α=0.10.

Methodologic Challenge: Alert Thresholds

Advantages of DELTA Surveillance

1. Automated Continuous, Prospective, Active Surveillance: suitable for monitoring of large clinical datasets using straightforward analytic methods. 2. Supports multiple, simultaneous, active surveillance analyses. 3. Pre-specified analytic plan: DELTA configured at outset of analysis to perform studies in accordance with SAP/protocol. 4. Propensity matching is uniquely suited to comparative safety

• analyses. Statistically robust, very easy to interpret and

explain, and is conducive to post-hoc analysis for signal exploration. 5. Pragmatic and Scalable Approach. Validated in central data model and distributed models.

25

Overview of DELTA

Frederic S. Resnic, MD MSc and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 3.61

• DELTA user interface, with integrated DELTAlytics

analytic engine, utilizing MySQL as back-end data repository

• Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTION…Site Admin Setup

• Advanced Features
• DELTAlytics statistical package interface setup – part of Installation
• Site / Organizations
• Users / Permissions/Roles (all Demo Users are Organization Admins)
• Projects / Groups
• Source Data Connections and Table Relationships (Rely on your data

managers)

• JDBC databases allowed – MSSQL, MySQL
• Requires Database source file – even a Flat File must be imported

to db

• Alerts and Notifications

Note: Topic detail not covered today

28

DELTA Study Life-cycle Stages

• Model Builder – data field definitions

and utilization characteristics

• Study Configuration – analysis method

parameters and data set extraction

• Results – table and graph visualization

Overview of DELTA System

Note: Topic detail not covered today

29

Requires Completed Data Source/Tables and Table Relationships

• Atomic Fields
• Field Creation – virtual (primary/secondary)
• Handling Missing
• Data Acquisition (Staging) and Extraction (Flat Table)
• Descriptive Statistics (Staging/Flat Table)

Note: Topic detail not covered today

Overview – Data Model Builder

30

Overview – Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview – Study Configuration

Requires Successful Model Processing

• Model Selection
• Identification of Study Dates, Case and Sequencer

Fields

• Analysis Method Selection
• Completion of Method Parameters
• Study Submission and Analytic Processing

Note: Topic detail not covered today

32

Overview –DELTA Methods

DELTA Statistical Methods

• Logistic Regression

(including LR Make and LR Apply)

• Propensity Analysis

(including PA Create, PA Match and PA Outcomes)

• Survival

(including standard and propensity score matched survival)

Note: Topic detail not covered today

33

Overview – Propensity Analysis Method

DELTA Propensity Analysis Method

• Matched propensity parameters
• Outcomes – single or multiple
• Covariate selection for match formula
• Match characteristics
• Options - Alpha Spending and Automated Variable Selection
• Method configuration divided as separate analysis steps:
• Create match formula – formula stored
• Match using formula– case set stored
• Outcomes – process outcomes of existing match set

34

Overview - Study Results

Results Visualization

• Dashboard
• Method specific Charts and Tables
• Export to .CSV, .PDF, .PNG, .JPEG

35

Overview - Study Results

List of PA chart and table visualizations:

• Risk Adjusted Observed/Expected Chart (Periodic)
• Event Rate Comparison Chart
• Risk Adjusted Observed/Expected Chart (Area)
• Event Rate Comparison Chart Data
• Observed Expected Chart Data
• Descriptive Statistics – Field stats
• Covariate Matched Statistics – Field stats Prior, After,

Unmatched

• Logistic Regression Formula Data
• Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

• 1. Evaluate Protocol required fields, including devices and
• utcomes
• 2. Evaluate Protocol method parameters
• 3. Prepare source database tables, define DELTA connections
• 4. Select Atomic fields, assign class/kind and process Acquistion
• 5. Create virtual fields, assign class/kind
• 6. Review Descriptive Stats, Impute and enter ‘Missing’
• 7. Process All fields
• 8. Create and Configure Studies, run Analyses

CathPCI DELTA Pilot

• Outcomes: In-hospital significant vascular and bleeding:
• Primary Safety: Any Vascular Complication – composite of access

site bleeding requiring treatment, large hematoma (>6cm), retroperitoneal hemorrhage, vascular comp. requiring intervention.

• Secondary Safety : Access site bleeding alone, requirement for

post-procedural blood transfusion.

• Pre-specified Analyses: per written SAP
• High risk populations: Women, diabetics, age≥70 years.
• Alternative risk adjustment: Multivariate logistic risk adjustment
• Falsification Hypothesis: Thrombectomy catheter adverse events

(anticipated no difference expected between devices).

• Interim Reviews: pre-specified (SAP) for review by oversight

committee after 12 months and at study completion.

CathPCI DELTA Pilot

39

Atomic and Virtual Fields – Outcome example

• Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication – 8035 PostBleedHematoma >5-10cm, > 20cm – 8060 PostRetroBleed – 8070 PostBleedAccessSite – 8055

• DELTA virtual field formula from Atomic Dichotomous fields

({PostOtherVascComp]=1) OR ({PostBleedHematoma}=1) OR ({PostRetroBleed}=1) OR ({PostBleedAccessSite}=1)

CathPCI DELTA Pilot

40

Partial List – Field Formulas

Field Description mysql_syntax Calculated BMI = Weight(kg)/(Height(m)*Height (m)) (Weight/(Height*Height)) Is BMI >=21 and <25 IF(t_BM1>=21.0 AND t_BM1<25.0, 1,0) Is BMI >=30 IF(t_BMI>=30.0, 1 ,0) Is BMI < 21.0 IF(t_BMI>21.0, 1,0) Is Age> 70 IF(Age >=70, 1 ,0) CAD Presentation (5000) = 'STEM! or Equivalent' IF({CADPresentation_TEXT}='STEMI or equivalent', 1 ,0) PCI Status 7020 IF(PCIStatus_TEXT='Emergency', 1,0) IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 1,1,0) 1 Immediate PCI for STEM! IF(PCindication=1, 1,0) MAX pre-procedure Lesion Severity Max(StenosisPriorTreat) Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI) IIF ({CIRCStenosis}>50, 1,0) IF(CIRCStenosis >=50, 1 ,0) IIF ({MidLADDz}= 1 or {ProxLADDz}= 1,1,0) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 1,1,0) IIF ({NumVessei_Dz}>3,3,{NumVessei_Dz]) IF(f_NumVessel_Dz >3,3,'f_NumVessei_Dz') Is any thrombectomy device class = Aspiration IF(ICDeviceClass='Aspiration',T,'F) Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass='Active' AND t_lsDEVFamily_Mynx<=> 1,1,0) Is any Vascular Closure Device Class= Active IF(ClosureDevClass='Active',T ,'F) Is any Closure Device Family Mynx IF(ClosureDevFamily='Mynx',T,'F) Is any MedUsed = Aspirin (any) IF(MedUsed}='Aspirin',T,'F) Has Any ln-Stent Thrombosis IF(InThrombosis= 1,'T','F') Has any previously treated lesion IF(PrevTreatedlesion= 1,T,'F) Presence of any graft lesion IF(LesionGraft=' 1; 2; 3','T','F') MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=1,1,0)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

• Propensity Match Analysis: 1:1 match of patients treated with

Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case.

• Non-parsimonious propensity model (likelihood of receiving

Mynx), fixed caliper width=0.01, greedy match.

• Quarterly analysis with adjustment for multiple comparisons

(O’Brien-Fleming) for each endpoint.

• Alert if 95% CI of differences between proportions (Wilson

method) do not cross zero.

• Missing Data: Simple imputation (unknown “missingness” in

advance)

• Post-Hoc Analyses: On basis of interim and final reviews, Scientific

Committee recommended sensitivity analyses and analysis of independent validation dataset (10/1/13-9/20/15)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45

Source: Resnic FS et al.

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

• Live Demo Site:

https://CERIdemo.lahey.org/DELTA3

• Download site:

https://bitbucket.org/CERI-Lahey/delta3/downloads/

• On-line documentation:

User Manual: https://ceri- lahey.atlassian.net/wiki/spaces/DUM/overview Statistical Methods: https://ceri- lahey.atlassian.net/wiki/spaces/DSMS/overview Questions?

51

DELTA Interactive DEMO and Questions

Fred Resnic, Henry Ssemaganda Susan Robbins

https://CERIdemo.lahey.org/DELTA3

Learning Objectives

• Understand basic fundamentals of DELTA
• Menus, features and functions
• Understand Data Model
• Atomic and virtual fields, missing substitutions,

data acquisition and extraction, descriptive statistics and flat table

• Understand Study Configuration
• Basic Propensity Analysis parameters
• Understand Results Visualization
• Dashboard and Charts/Tables

53

• 1. Model review
• 2. Atomic fields, assign class/kind
• 3. Create virtual fields, assign class/kind
• 4. Create and Configure Study
• 5. Review Demo Results

You can try cloning a model, editing it and processing… And try cloning a study, assign new model, change a parameter and submit…we’ll see how you do!!!

DELTA Demo - Topics

54

DELTA Demo – Login

• Login with Username/Password on card
• Brief intro to Projects, Feedback, Stat Program,

User Dropdown buttons

• DELTA Menu

Select “Data Models”

55

DELTA Demo – Models

Sample Model Grid

• Review Step 1: Manage (Add, Edit, Clone,

Delete)

• Step 2: Build Model Structure

Select (“Demo Model”) and click Step 2

• Model Builder
• Review Step 1A, 1B, 1C

(Source/Atomic/Relationships)

56

DELTA Demo – Model Source Data

• Step 1A – Select well structured dataset

Normalized with Primary Key Can manage 1:many with Step 1C ‘Relationships’ Sample Data is a Flat Table in database

• Step 1B – Atomic Field Selection

(click +) to display fields, check mark indicates those added to the model

Note: Topic detail not covered today

57

DELTA Demo – Model Field Grid

• Atomic – from Source

(Type read from Source – will prompt if not valid in DELTA)

• Virtual – Primary are based on Source Column
• Virtual – Secondary are based on Primary Virtual

(stacked)

• Missing – enter formula/value on Atomic AND

flag in “Missing” column (Verify formulas before processing)

• Field Characteristics
• Class/Kind relate to USE of data by DELTA

58

DELTA Demo – Model Data

Sample Model Grid

• Review Step 2 and 3: Acquisition, Virtual Fields,

Extraction “STATUS: Processing of Descriptive Stats finished” Select Step 3: Data Extraction > All Statistics

• Review Statistics table

59

DELTA Demo – Study Configuration

• Review Step 1: Add Study
• Review Step 2: Edit Study
• Review Study Grid fields

Select (“PA Demo Study 2”) and click “Step 2” Then “Edit parameters”

Select “Studies”

60

Note: Can schedule to rerun without intervention

• Study Grid Fields

DELTA Demo – Study Configuration

Parameters

61

DELTA Demo – Study Configuration

• Sample Study PA Method Parameters

62

DELTA Demo – Study Results

• Sample Result (PA Demo Study 2 P1)
• Review ‘Result Dashboard’
• Review ‘Result Charts’

Select (“PA Demo Study 2 P1”) and click “Dashboard” and/or “Result Charts”

Select “Results”

Thank You

Lahey Clinic Medical Center Susan Robbins, BS Henry Ssemaganda, MD, MPH Marek Mizeracki, MSc Vanderbilt University VAMC Michael Matheny, MD MSc MPH Dax Westerman, MSc For more information please contact: Frederic S. Resnic, MD MSc Lahey Hospital and Medical Center 41 Mall Road Burlington, MA 01805 781-744-2778 frederic.resnic@lahey.org

64

DELTA Administration Overview

• Organizations
• Users
• Permissions and Roles
• Projects/Groups
• Database connections
• Statistical Method Interface

65

DELTA Administration