PCORI Evaluation Group Tenth Meeting November 5, 2014 Agenda for - - PowerPoint PPT Presentation

▶
pcori evaluation group
SMART_READER_LITE
LIVE PREVIEW

PCORI Evaluation Group Tenth Meeting November 5, 2014 Agenda for - - PowerPoint PPT Presentation

Oct 21, 2023 375 likes •803 views

PCORI Evaluation Group Tenth Meeting November 5, 2014 Agenda for Today Recap from 10/15 meeting: Status updates Communicating about PCORI evaluation work Focus on topic capture and research prioritization Process to date

slide-1
SLIDE 1

PCORI Evaluation Group

Tenth Meeting November 5, 2014

slide-2
SLIDE 2

Agenda for Today

Recap from 10/15 meeting:

  • Status updates
  • Communicating about PCORI evaluation work

Focus on topic capture and research prioritization

  • Process to date
  • Evaluation planning

Key take-away points

2 PEG Meeting, November 5, 2014

slide-3
SLIDE 3

3

Dec 2013 In-person Kick-off:

  • Brainstorming and prioritizing

evaluation questions

  • Metrics for strategic goals

Jan 2014:

  • Measuring engagement and

its impact in PCORI projects Feb 2014:

  • Metrics for strategic goals
  • Evaluation Framework and

prioritization of evaluation questions March 2014:

  • Measuring engagement and

its impact in PCORI projects

  • CER Surveys: researchers,

patients, & clinicians April 2014:

  • Measuring goal #1:

useful information May 2014:

  • Overview of current data

collection plans

  • CER survey: Researchers

June 2014:

  • PCORI Dashboard:

metrics & visuals July 2014:

  • Need for an external evaluation for
  • verall impact of PCORI?
  • Methods for evaluating merit

review October 2014:

  • Communicating PCORI’s

Evaluation Activities

  • Lessons learned for evaluation:

Pronovost & Jha, 2014 (NEJM)

HISTORY OF THE PCORI EVALUATION GROUP

PEG Meeting, November 5, 2014

slide-4
SLIDE 4

PCORI Evaluation Goals

Steering PCORI: Determine progress against 3 PCORI goals Provide the public a framework for evaluating PCORI progress and provide progress updates Meet GAO requests

4 PEG Meeting, November 5, 2014

slide-5
SLIDE 5

Status update: Measuring Progress on PCORI Strategic Goals

Goal 1: Useful information

  • Applying usefulness criteria to the CER portfolio

Goal 2: Uptake of information

  • Tracking early indicators of dissemination

Goal 3: Influence research

  • Building repository of examples of PCORI’s influence

Future activities:

  • Track additional metrics as study findings are available

and implemented

PEG Meeting, November 5, 2014 5

slide-6
SLIDE 6

Status update: PCORI Evaluation Framework

Evaluation questions, metrics, and data sources

  • utlined and prioritized to guide evaluation projects
  • What is the impact of PCORI’s approach to Topic

Generation and Research Prioritization? discuss at this meeting

  • What is the Impact of PCORI’s Approach to

Communication, Dissemination, and Implementation

  • f information from funded research?  to discuss

after release of PCORI’s D&I framework (Dec 2014)

6 PEG Meeting, November 5, 2014

slide-7
SLIDE 7

Status Update: PCORI Dashboard

Used for quarterly reporting to PCORI Board of Governors

  • Improvements in data to populate the dashboard and the

visual presentation

Future activities:

  • Update dashboard as more advanced metrics of PCORI

strategic goals are available

PEG Meeting, November 5, 2014 7

slide-8
SLIDE 8

PCORI Data Collection and Evaluation

8

Impact of PCORI Portfolio Merit Review Best Practices in Research Engagement Patient and Other Stakeholder Knowledge, Attitudes and Behavior PCORI Events

PEG Meeting, November 5, 2014

slide-9
SLIDE 9

Agenda for Today

Recap from 10/15 meeting:

  • Status updates
  • Communicating about PCORI evaluation work

Focus on topic capture and research prioritization

  • Process to date
  • Evaluation planning

9 PEG Meeting, November 5, 2014

slide-10
SLIDE 10

Topic Capture and Research Prioritization Evaluation Questions

  • 1. What is the impact of PCORI’s approach to Topic Generation,

Prioritization, and Selection (inclusion of patients and other stakeholders, methods for ranking and selection) on:

  • perspectives incorporated into topic selection process,
  • the topics selected for funding, and
  • PCORI projects filling identified research gaps?
  • 2. Compared to broad funding announcements, what is the effect
  • f targeted funding announcements on the impact of

information?

  • 3. Compared to funding opportunities developed with input from

scientists only, what is the effect of funding opportunities developed based on multi-stakeholder input on the impact of information?

PEG Meeting, November 5, 2014 10

slide-11
SLIDE 11

Model: Topic Capture and Research Prioritization Evaluation Questions

PEG Meeting, November 5, 2014 11

slide-12
SLIDE 12

Evaluation questions: 1 of 3

  • 1. What is the impact of PCORI’s approach to

Topic Generation, Prioritization, and Selection on:

perspectives incorporated into topic selection process:

  • Describe perspectives captured through public topic

solicitation and engagement outreach

  • Tracking of progress of topics through prioritization

by stakeholder category

  • Comparison of submissions against funded portfolio,

by stakeholder category

PEG Meeting, November 5, 2014 12

slide-13
SLIDE 13

Hearing from stakeholders PCORI Engagement Workshop November 2012

Patients don’t have research questions, they have questions There are research questions others have identified that PCORI should address Provide tracking of progress of topics through prioritization

PEG Meeting, November 5, 2014 13

slide-14
SLIDE 14

The Advisory Panel meeting gave me the

  • pportunity to provide input on PCORI's

research topics

14

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree Apr-13 Jan-14

slide-15
SLIDE 15

The research prioritization process allowed me to systematically rank research topics

15

0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 45.00% 50.00% Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree Apr-13 Jan-14

slide-16
SLIDE 16

I was able to objectively prioritize the research topics, without giving special preference to topics that are more relevant in my professional or personal life

16

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree Apr-13 Jan-14

slide-17
SLIDE 17

PCORI's method for research topic prioritization will help PCORI fund research that can inform health care decisions by patients

17

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree Apr-13 Jan-14

slide-18
SLIDE 18

Evaluation questions: 1 of 3

  • 1. What is the impact of PCORI’s approach on:
  • perspectives incorporated into topic selection

process,

  • the topics selected for funding, and
  • PCORI projects filling identified research gaps?
  • Portfolio comparison PCORI vs …NIH
  • Stakeholder rating of topic relevance
  • Usefulness to end users

PEG Meeting, November 5, 2014 18

slide-19
SLIDE 19

Evaluation Questions: 2 of 3

  • 2. Compared to broad funding

announcements, what is the effect of targeted funding announcements on the impact of information?

  • Stakeholder rating
  • Information use

PEG Meeting, November 5, 2014 19

slide-20
SLIDE 20

PEG Meeting, November 5, 2014 20

slide-21
SLIDE 21

Establishing metrics: Evaluation questions

Is topic capture from the public yielding targeted PFAs and funded research proposals? How well does the PCORI research prioritization process address research gaps identified by scientists, patients, and stakeholders?

PEG Meeting, November 5, 2014 21

slide-22
SLIDE 22

22

0.67% 7.04% 1.09% 6.95% 10.39% 2.93% 10.05% 7.29% 8.04% 16.33% 0.50% 1.34% 2.60% 0.42% 0.34% 0.25% 0.50% 9.21% 9.55% 0.08% 1.84% Urinary Disorders Trauma/Injury Skin Diseases Respiratory Diseases Reproductive and Perinatal Health Rare Diseases Nutritional and Metabolic Disorders Nervous System Disorders Muscular and Skeletal Disorders Mental/Behavioral Health Liver Disease Kidney Disease Infectious Diseases Eye Diseases Ear, Nose and Throat Diseases Digestive Disorders Dental Health Cardiovascular Health Cancer Blood Disorders Allergies & Immune Disorders

Submitted Topics by Disease/Condition*

*Excludes topics where the disease/condition is “Unspecified”

PEG Meeting, November 5, 2014

slide-23
SLIDE 23

23

5 10 15 20 25 30 35 40 Trauma/Injury Skin Diseases Respiratory Diseases Reproductive and Perinatal Health Rare Diseases Other or Non-Disease Specific Nutritional and Metabolic Disorders Neurological Disorders Muscular and Skeletal Disorders Multiple/co-morbid chronic conditions Mental/Behavioral Health Liver Disease Kidney Disease Infectious Diseases Digestive System Diseases Cardiovascular Health Cancer Allergies and Immune Disorders NUMBER OF FUNDED PROJECTS

FUNDED PROJECTS BY DISEASE/CONDITION AND PROGRAM AREA (N=222)*

AD APDTO CDR IHS

PEG Meeting, November 5, 2014 *excludes all Methods projects

slide-24
SLIDE 24

24

0.00% 2.00% 4.00% 6.00% 8.00% 10.00% 12.00% 14.00% 16.00% 18.00% % of Total

Submitted Topics (n=1163) v Funded Projects (n=189) by Disease/Condition*

Submitted Topics Funded Projects

PEG Meeting, November 5, 2014

*For topics, this does not include topics which do not specify a disease/condition. For projects, this does not include projects which are Methods, non-disease specific, or deal with multiple chronic conditions.

slide-25
SLIDE 25

2 4 6 8 10 12 14

Allergies and Immune Disorders Blood Disorders Cancer Cardiovascular Health Dental Health Dermatology Digestive Disorders Ear, Nose and Throat Disorders Eye Diseases Genetic Disorders and Rare Disease Infectious Diseases Kidney Disease Liver Diseases Mental Health Muscular and Skeletal Disorders Neurological Disorders Nutritional and Metabolic Disorders Reproductive and Perinatal Care Respiratory Disorders Trauma Urinary Disorders

Submitted topics (%) Prioritized Topics (%)

*Excludes ‘Unspecified’ topics (submitted N=556; prioritized N=13)

Spring 2013 Submitted Topics (N=923) & Prioritized Topics by Condition (N=33)*

slide-26
SLIDE 26

Comparison of submitted topics to IOM 100

EXACT MATCH

  • The suggestion has the exact (or

nearly exact) wording of the IOM 100,

  • r the suggestion includes both the

specific condition and intervention/comparators of an IOM 100.

SIMILAR

  • The suggestion matches an IOM 100

disease/condition(s), or matches an IOM 100 intervention.

DISSIMILAR

  • The suggestion does not match any of

the disease/conditions or interventions

  • f the IOM 100.

5.2 16.6 78.3 Exact Similar Not Similar

% Matching IOM 100 Topics

PEG Meeting, November 5, 2014 26

slide-27
SLIDE 27

Submitted Topics Matching IOM 100 Priorities by Condition (N=1200)*

10 20 30 40 50 60 70 80 90 100

Urinary Disorders Trauma/Injury Respiratory Diseases Reproductive/Perinatal Health Nutritional/Metabolic Disorders Nervous System Disorders Muscular/Skeletal Health Mental/Behavioral Health Liver Diseases Kidney Disease Infectious Diseases Genetic Disorders/Rare Diseases Eye Diseases Ear/Nose/Throat Diseases Digestive Disorders Dermatology Dental Health Cardiovascular Health Cancer Blood Disorders Allergies/Immune Disorders

% Exact Match % Similar Match % Not Similar

*excludes ‘Unspecified’ topics N=603

slide-28
SLIDE 28

Participating in the Advisory Panel meeting helped me to understand other peoples' perspectives on research

28

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree

Apr-13*

Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree

*No data available for Jan-14

slide-29
SLIDE 29

The Advisory Panel meeting gave me the

  • pportunity to provide input on PCORI's

research topics

29

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree Apr-13 Jan-14

slide-30
SLIDE 30

PCORI's method for research topic prioritization will help PCORI fund research that can inform health care decisions by patients

30

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree Apr-13 Jan-14

slide-31
SLIDE 31

April 2013 Prioritization by Stakeholder APDTO Panel (1 of 2)

31

2 4 6 8 10 12 Patients Clinicians Researchers Other Stakeholders Total Ranking

slide-32
SLIDE 32

April 2013 Prioritization by Stakeholder APDTO Panel (2 of 2)

32

2 4 6 8 10 12 14 16 18 20 Patients Clinicians Researchers Other Stakeholders Total Ranking

slide-33
SLIDE 33

Key Take-Away Points

PCORI is evaluating topic capture and research prioritization along with other PCORI work. We are looking at the:

  • types of stakeholders that submit topics
  • contribution to prioritization by stakeholder type
  • comparison of submitted topics to PCORI-funded topics.

What other information should we collect to evaluate the impact of our topic capture and research prioritization process?

PEG Meeting, November 5, 2014 33

slide-34
SLIDE 34

Appendix

PEG Meeting, November 5, 2014 34

slide-35
SLIDE 35

Goal #1: Usefulness Criteria

35

User-Driven

  • The end-users (patients, clinicians, payers, organizations, health systems etc.) of the information

have been identified (e.g., in the literature, through engagement with partners).

  • The end-users (patients, clinicians, payers, organizations, health systems etc.) have identified

this information would fill a critical gap (e.g., end-users generated the research questions).

  • The end-users have committed to using the information (e.g., systems

administrators/clinicians/etc. have committed to implement the intervention)

User-Focused

  • The research assesses options that are relevant for the end users of the information.
  • The end-users were involved in choosing or developing the options.
  • The research assesses the outcome(s) that will comprehensively address the needs of the end-

users.

Real-World Users

  • Results can provide a clinically (in addition to statistically) significant answer. The study would

provide a clear answer, rather than calling for further research.

  • Results can inform decisions of end-user(s) with specific characteristics, conditions, and

preferences.

  • Results can be scaled/spread beyond the traditional study setting for a wider net impact.

PEG Meeting, November 5, 2014

slide-36
SLIDE 36

Goal #2: Uptake

Dissemination

(Measure for all PCORI funded studies)

Uptake and Use

(Measure for a subset of PCORI funded studies)

Impact: Changes in Health Decisions or Care and Outcomes

(Measure for small set of exemplar studies)

36

  • Results reported back to study

participants

  • Access to PCORI study report
  • Presentations:
  • Scientific/professional audiences
  • Lay audiences
  • Bibliometrics:
  • # of Publications
  • Time to publication
  • Impact factor
  • Citations
  • Alternative metrics for key groups

(patients, clinicians, payers, etc.):

  • # manuscript downloads
  • # manuscript bookmarks
  • Media coverage
  • Social media coverage
  • Adoption of study findings in the

study setting

  • Incorporation into:
  • Systematic reviews
  • Patient and consumer education

materials

  • Graduate Medical Education

(GME) or Continuing Medical Education (CME)

  • Practice guidelines
  • Decision making infrastructure

(e.g. electronic decision aids, clinical reference tools)

  • Payer policies
  • Institutional, local, state, and

national policy

  • Improvement in health

decisions or health care quality and improved health outcomes in relevant populations

Note: Most of these metrics are typically not measurable until after study completion, and in many cases, are typically not measureable until several years after study completion.

PEG Meeting, November 5, 2014

slide-37
SLIDE 37

Goal #3: Influence

We are now or soon will be measuring:

  • Endorsement, Promotion, and Dissemination of PCORI work
  • Use of PCORI Methodology Standards for Patient-Centeredness
  • Use of PCORI approaches:
  • Topic Generation and Research Prioritization
  • Merit Review
  • Engagement
  • Communication and Dissemination
  • Use of PCORI guidance re: Patient-Centered CER
  • Use of PCORI-supported curricula or training
  • Collaborations/Co-funding with other funders

We will have to wait a few more years to measure:

  • Use of PCOR Methods evidence
  • Use and support of PCORnet

37 PEG Meeting, November 5, 2014

slide-38
SLIDE 38

Evaluation Framework

38 PEG Meeting, November 5, 2014

slide-39
SLIDE 39

Topic Capture and Targeted Funding

Falls in the Elderly: Clinical Trial of a Multifactorial Fall Injury Prevention Strategy in Older Persons Treatment Options for Severe Asthma in African- Americans and Hispanics & Latinos Comparative Effectiveness Research on Medical and Surgical Treatment for Uterine Fibroids Obesity Treatment Options Set in Primary Care For Underserved Populations The Effectiveness of Transitional Care

39 PEG Meeting, November 5, 2014

slide-40
SLIDE 40

Topics prioritized 1-5 by Advisory Panels, received pragmatic trial application

Identifying lung cancer in people with lung nodules Ductal Carcinoma Interventions to Promote Tobacco Cessation Among Vulnerable Populations Integration of Mental and Behavioral Health Services into Primary Care Settings Care management of multiple chronic conditions Hepatitis C Treatment options for opioid substance abuse Migraine headaches Coronary Artery Disease Treatment options for autism Osteoarthritis Heart attacks among racial and ethnic minorities Proton beam therapy for breast, lung, and prostate cancer

40 PEG Meeting, November 5, 2014

slide-41
SLIDE 41

Topics prioritized 1-3 by Advisory Panels, prioritized by at least 2 other stakeholders (e.g. IOM, AHIP)

Ductal Carcinoma Integration of mental and behavioral health services into primary care settings Cancer management Palliative care management Treatment Strategies for Atrial Fibrillation Migraine headache Coronary Artery Disease Major depressive disorders Treatment options for autism Heart attacks among racial and ethnic minorities

41 PEG Meeting, November 5, 2014