Data and Safety Monitoring in Pragmatic Trials Greg Simon Outline - - PowerPoint PPT Presentation

Greg Simon

Data and Safety Monitoring in Pragmatic Trials

Outline

• Distinguish specific questions
• For each question:
• Describe goals and process of monitoring
• Describe what’s different about pragmatic trials

2

What are we monitoring?

From 1998 NIH policy: “Evaluate the progress of interventional trial(s), including periodic assessments of data quality and timeliness, participant recruitment, accrual and retention, participant risk versus benefit, performance of trial sites, and other factors that can affect study outcome.”

3

What are we monitoring?

• Viability – Are we recruiting enough of the right kind of

people?

• Fidelity – Are treatments/programs being implemented or

delivered adequately?

• Adverse Events – Are study treatments or procedures

causing harm?

• Safe Practice – Are study staff providing safe and

appropriate care in high-risk situations?

• Benefit – Do we already know which treatment is superior?

4

Viability

• Why– Will the sample be adequate to answer the question?
• What – Monitor overall rate of recruitment and

characteristics of those recruited

• How – Compare recruitment rate and sample characteristics

to assumptions used for power calculations

• When – Throughout recruitment period – but especially early

in recruitment.

• Who – Can assess without knowing treatment assignment.

Study team, funding agency, and DSMB can see same data.

5

Viability – What’s different in pragmatic trials?

• If recruitment is more automated (i.e. less dependent on

provider referral), rate may be more predictable.

• But – if recruitment is limited to specific practice settings,

increasing recruitment may be more difficult.

• Generalizability may be more important.

6

Fidelity / Adherence

• Why– Will the “separation” between study arms allow a valid

test of the study question?

• What – Summary measures of quality or fidelity of treatment

delivery, focusing on key differences between study arms.

• How – Compare “separation” to assumptions used for power

calculations; Examine contamination or cross-over.

• When – Throughout intervention period.
• Who – Depending on design specifics, DSMB and study

team may or may not be able to see the same data.

7

Fidelity/Adherence – What’s different in pragmatic trials?

• Need to be clear whether study question primarily concerns

efficacy, effectiveness, or implementation.

• Tension between maximizing “separation” and

generalizability.

8

Individual Adverse Events

• Why– Identify unanticipated harms of study procedures or

treatments (signal detection).

• What – Case reports of adverse events, with enough detail

to determine attribution.

• How – Determine if individual events could be attributable to

study procedures or interventions.

• When – Throughout intervention period.
• Who – May require breaking of blind, usually limited to

DSMB.

9

Individual Adverse Events – What’s different in pragmatic trials?

• Treatments are established and risks often well known.
• Attribution of “relatedness” for individual events may be

more difficult (if not impossible).

• Must often consider competing risks (especially for complex

interventions and/or patients with co-occurring conditions).

• Should we just stop doing this?

10

Rates of Adverse Events

• Why– Compare rates of anticipated harms of study

procedures or treatments (hypothesis testing).

• What – Rates of specific and/or overall adverse events.
• How – Compare rates (with appropriate caution for multiple

comparisons and sequential testing).

• When – Throughout intervention period.
• Who – Requires breaking of blind, usually limited to DSMB.

11

Rates of Adverse Events – What’s different in pragmatic trials?

• Treatments are established and risks often well known.
• Must often consider competing risks.
• Longer follow-up periods: Must consider differences in

timing for benefits and adverse events by intervention condition.

• What if the “adverse event” is the study outcome?

12

Safe Practice

• Why– Study staff assume some level of clinical

responsibility, creating the potential for conflicting interests.

• What – Reports regarding care provide in specific scenarios
• f concern.
• How – Evaluation of care provided against community

standards or standards established by protocol.

• When – Throughout intervention period.
• Who – May require breaking of blind, usually limited to

DSMB.

13

Safe practice – What’s different in pragmatic trials?

• Study staff often less directly involved in care.
• Information regarding concerning situations may be delayed

and limited in detail.

14

Benefit

• Why– Accelerate access to more effective treatments (and

minimize exposure to less effective ones).

• What – Interim data regarding study outcome(s).
• How – Sequential testing in comparison to a boundary or

stopping rule.

• When – Throughout follow-up period (but less important

early on).

• Who – Requires breaking of blind, usually limited to DSMB.

15

“Detectable Difference” threshold (for power calcs and interim analyses)

• General principle: What is the difference we would not want

to miss?

• For efficacy trials: Clinically meaningful difference at the

patient level - What difference would be large enough to affect a clinical decision?

• For pragmatic trials: Actionable difference at the population

level - What difference would be large enough to prompt implementation or change in policy?

16

Why stop? – Levels of ethical

• bligation
• Strong – How would stopping now affect people enrolled in

this trial?

• Moderate – How would stopping now affect other people

with this health conditions?

• Weak – How would stopping affect the broader community

(e.g. in terms of other uses for limited resources)?

17

What to stop?

Distinguish between:

• Not enrolling new participants
• Stopping delivery of a study treatment
• Disclosing results and allowing choice

Always depends on the specifics of the situation

18

Susan S. Ellenberg, Ph.D. Perelman School of Medicine Universit y of Pennsylvania

DATA AND SAFETY MONI TORI NG I N PRAGMATI C TRI ALS: PART 2

2

WHAT PCTs NEED A DMC?

wAn independent DMC is usually needed when

―Treat ment s and/ or disease are high risk ―Saf et y assessment will require comparison of

• ut comes by t reat ment group

―Credibilit y of result s part icularly import ant

wMost PCTs will probably need a DMC

―Will address issues t hat af f ect large populat ions ―May be int ended t o inf luence pract ice ―Result s may be subj ect t o int ense scrut iny

wBut some may not

―I f no saf et y imperat ive t o compare out comes during t rial, may not need a DMC

3

WHAT DATA NEED TO BE MONI TORED?

wShould adherence t o assigned t reat ment be monit ored?

―NO: pragmat ic t rials seek real world answers, so we want t o see what happens in act ual pract ice ―YES: import ant t o int erpret at ion of f indings; need t o disent angle adherence issues f rom t rue t reat ment ef f ect s

wShould a DMC make recommendat ions f or ways t o improve adherence?

―NO: again, need real world answer ―YES: lack of adherence may be due t o incomplet e underst anding of int ent of st udy

4

WHAT DATA NEED TO BE MONI TORED?

wI n t radit ional t rials, dat a qualit y is t ypically monit ored by t he DMC wOne aspect of dat a qualit y is care in ent ering

• nly part icipant s who meet inclusion crit eria

wI n some cases, when t rial is not double-blind, “ineligible” could be euphemism f or “participant doesn’t want this treatment,” or “I don’t want this participant to get this t reat ment ” wI mport ant t o monit or ineligibilit y rat es t o see if t reat ment groups dif f er

5

WHAT DATA NEED TO BE MONI TORED?

wFor clust er-randomized t rials, design of t en used in pragmat ic t rials, also import ant t o monit or t he “design f act or”

―I nt ra-clust er correlat ion coef f icient (I CC)—t he ext ent t o which result s wit hin a clust er will be more similar t han result s across clust ers—is a component

• f sample size calculat ion

―Typically, hard t o est imat e I CC f rom prior dat a ―I nt erim est imat es of I CC import ant t o see whet her st udy will have expect ed power

5

6

WHO SHOULD BE DOI NG THE MONI TORI NG?

wTradit ional DMC members: clinicians, biost at ist ician(s) ―Somet imes bioet hicist s ―Somet imes pat ient represent at ives wPragmat ic t rials may need special expert ise ―Pat ient reps may be more import ant ―May need communit y-based in addit ion t o academic clinicians ―For t rials deriving dat a f rom elect ronic healt h records, may need someone wit h expert ise in medical inf ormat ics

6

7

PATI ENT REPRESENTATI VES

• I ncluded on many DMCs f or NI H t rials
• Would seem especially valuable f or t rials wit h

pat ient -cent ered out comes

• Unique insight s
• Evaluat ing part icipant burden
• Balance of pot ent ial benef it s and harms
• What t ype of pat ient represent at ive?
• Scient ist who is also a pat ient ?
• Leader in pat ient advocacy organizat ion?
• Need f or all DMC members t o have a basic

underst anding of clinical t rials met hods, and appreciat e import ance of conf ident ialit y

8

MEDI CAL I NFORMATI CS

wPragmat ic t rials may increasingly derive dat a f rom elect ronic healt h records (EHRs) wMay involve more t han one EHR syst em wDif f erent syst ems may have dif f erent schedules f or updat ing f iles wOt her “new” t ypes of dat a, such as biosensors and act ivit y monit ors wComplexit ies in such dat a may require input

• f someone wit h more “high t ech” expert ise

9

CHALLENGES I N MONI TORI NG I NTERI M DATA

wOperat ional procedures may not be f ully st andardized across sit es, t o best ref lect “real world” pract ice wThis could mean in some cases t hat dat a will be collect ed on nonunif orm schedules wI nt erim comparisons of st udy out comes will need t o t ake t his complicat ing f act or int o account

9

1 0

MAKI NG DECI SI ONS ABOUT MONI TORI NG

wThe DMC and t he st udy sponsor and invest igat ors need t o reach consensus about monit oring approaches prior t o st udy st art wConsiderat ion of t he dimensions of “pragmat ic-ness,” as can be done f rom t he PRECI S crit eria, may f acilit at e t hese decisions