Dashboard Review Third Quarter of FY 2014 Joe Selby, MD, MPH - - PowerPoint PPT Presentation

▶
dashboard review third quarter of fy 2014
SMART_READER_LITE
LIVE PREVIEW

Dashboard Review Third Quarter of FY 2014 Joe Selby, MD, MPH - - PowerPoint PPT Presentation

Feb 12, 2024 215 likes •560 views

Dashboard Review Third Quarter of FY 2014 Joe Selby, MD, MPH Executive Director Board of Governors Meeting, September 15, 2014 21 First Message from the Dashboard: Notice the Colors Legend For those of us who cant see On Target red,

slide-1
SLIDE 1

Dashboard Review Third Quarter of FY 2014

Joe Selby, MD, MPH Executive Director

Board of Governors Meeting, September 15, 2014 21

slide-2
SLIDE 2

First Message from the Dashboard: Notice the Colors

Board of Governors Meeting, September 15, 2014 22

Needs Attention On Target Off Target Legend

For those of us who can’t see red, the red boxes would show up in gray scale as dotted

slide-3
SLIDE 3

20 40 60 80 100 Award to Contract Contracts Response Science Response Percent of Target

w/in 90 cal. days w/in 2 bus. days w/in 3 bus. days

Q1 Q2 Q3 Q4

Ambassadors Fully Trained

Q3

50 100 150 200 250 300 FY 2012 FY 2013 FY 2014 $M Targeted Broad PCORnet Pilots

10 20 30 40

Cycle I Cycle II Cycle III August 2013

Percent % of Applications Awarded % of Resubmissions Awarded

NA

Needs Attention On Target Off Target

Legend Broad Awards Success Rates Dissemination and Uptake

Board of Governors FY2014 Dashboard – Q3 (As of 6/30/2014) Our Goals: Increase Information, Speed Implementation, and Influence Research

*Influencing Research* PCORI is the catalyst for a proposal to establish the Center for Pediatric Comparative Effectiveness Research at the University of Arkansas for Medical Sciences

20 40 60 80 100 Research P2P PPRNs CDRNs Percent Meeting All

Q1 Q2 Q3 Q4

Percent of Projects Meeting All Milestones Completion of Phase I of PCORnet

Q2

Funded Research Portfolio Impact of Engagement on Merit Review 2014 Research Funding Commitment

Q3

Responsiveness

$528M Q1 Q3 100

Current Actual Target Previous Actual

Pipeline to Proposal Awards

Q4 65 1/3

  • f

Phase I Q3

2014 Expenditures on Research Awards – $M

Q1 $94M

30 X

Expense Categories External Research Commitments

20 40 60 80 100 FY 2014 FY 2015 Percent Program Program Support Admin Actual Budget 62

Q2 Q3

20 40 60 80 100

FY 2013 FY 2014 All All by $

Percent Other Screening Treatment Diagnosis Prevention

NA

Other 2014 Expenditures – $M

Q2 Q1 $89M Q3

2014 Staffing Plan – Number of People

Q1 Q2 Q3 164

NA=Not Applicable

NA NA 63 136 NA NA NA 30 191 20 40 60 80 100 Journal Articles Number 1,000 2,000 3,000 4,000 5,000 6,000 7,000 Web Views 2 4 6 8 10 Citations

Q1 Q2 Q3 Q4 (of Methodology Standards)

Priority Topics to Be Awarded in 2014

Q3 5

2

and

Q1

40 60 80 100 Cycle II Cycle III Aug 13 Winter 14

Percent of reviewers who think patient & stakeholder reviewers provide valuable input

slide-4
SLIDE 4

Noteworthy Items

Board of Governors Meeting, September 15, 2014 24

slide-5
SLIDE 5

Influence Research

Early Signs

  • Dr. Debra Fiser was a critical care physician for 14 years, then Chair of the University of

Arkansas for Medical Sciences (UAMS) Department of Pediatrics for 11 years, then served as Dean of the UAMS College of Medicine for 7 years. She said as a capstone for her career, she wanted to help others in her department achieve success by mentoring young clinicians and

  • faculty. She particularly wanted to help physicians who have not been involved in research in

the past to see the relevance of research in their work and to build capacity to engage in Comparative Effectiveness Research (CER).

  • Dr. Fiser stepped down as Dean in May 2013 and attended the PCORI Engagement Workshop

in Memphis, Tennessee, in August of 2013, which provided inspiration for her proposal to establish a Center for Pediatric Comparative Effectiveness Research at UAMS. She saw the work PCORI is doing to fund patient-centered CER as the catalyst UAMS needed to invest in training faculty in CER. The objective of the Center is to recruit, train, and mentor faculty in patient-centered CER

  • methods. It is up and running as of July 1, 2014, and currently has funding for five years to

recruit five half-time faculty each with three years of support. The Center is focused on building capacity by preparing pediatric clinicians and faculty who are relatively new to research to conduct patient-centered CER, with an end goal of successfully applying to PCORI for funding.

Board of Governors Meeting, September 15, 2014 25

*Influencing Research* PCORI is the catalyst for a proposal to establish the Center for Pediatric Comparative Effectiveness Research at the University of Arkansas for Medical Sciences

slide-6
SLIDE 6

Increase Information

Merit Reviewer Survey Details

26

Survey Sample Size Survey Completion Rate Percent Who Agree Somewhat or Strongly Cycle II

205 94% 89%

Cycle III

167 97% 85%

August 2013

286 91% 93%

Winter 2014

209 83% 94%

Surveys of merit reviewers are conducted immediately following each in-person merit review and are live in the field for 2 weeks.

Board of Governors Meeting, September 15, 2014

We asked merit reviewers to rate their agreement with the following statement: The patient/stakeholder reviewers provided valuable input during the discussion

(Strongly Disagree, Somewhat Disagree, Neither Agree nor Disagree, Somewhat Agree, and Strongly Agree)

slide-7
SLIDE 7

Percent of merit reviewers who agree with the statement: “The patient/stakeholder reviewers provided valuable input during the discussion”

Board of Governors Meeting, September 15, 2014 27

71% 85% 82% 89% 80% 81% 90% 96% 56% 56% 69% 75% 24% 6% 16% 3% 12% 19% 10% 2% 29% 23% 20% 17% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Cycle 2 Cycle 3 August 2013 Winter 2014 Cycle 2 Cycle 3 August 2013 Winter 2014 Cycle 2 Cycle 3 August 2013 Winter 2014 Patients Stakeholders Scientists

% Somewhat agree % Strongly agree

slide-8
SLIDE 8

Percent of merit reviewers who agree with the statement: “The scientific reviewers provided valuable input during the discussion”

Board of Governors Meeting, September 15, 2014 28 73% 82% 84% 89% 82% 83% 90% 83% 79% 76% 84% 85% 15% 9% 13% 6% 13% 17% 8% 15% 18% 18% 11% 12%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Cycle II Cycle III August 2013 Winter 2014 Cycle II Cycle III August 2013 Winter 2014 Cycle II Cycle III August 2013 Winter 2014 Patients Stakeholders Scientists

% Somewhat agree % Strongly agree

slide-9
SLIDE 9

Percent of merit reviewers who agree with the statement: “Overall, scientific reviewers were receptive to input from patient and stakeholder reviewers”

Board of Governors Meeting, September 15, 2014 29 51% 68% 63% 78% 58% 69% 78% 72% 64% 73% 81% 83% 24% 18% 29% 17% 32% 17% 16% 17% 32% 21% 16% 15%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Cycle II Cycle III August 2013 Winter 2014 Cycle II Cycle III August 2013 Winter 2014 Cycle II Cycle III August 2013 Winter 2014 Patients Stakeholders Scientists

% Somewhat agree % Strongly agree

slide-10
SLIDE 10

Increase Information

Our first published study of Merit Review

Board of Governors Meeting, September 15, 2014 30

slide-11
SLIDE 11

Influence Research PCORI Ambassadors Program

31 Board of Governors Meeting, September 15, 2014

Ambassadors Enrolled by Stakeholder Group

(Cumulative)

5 10 15 20 25 30 Q1 Q2 Q3

West: 11 Southwest: 15 Midwest: 18

Ambassadors Enrolled by Geographic Region

Southeast: 26 Northeast: 30

slide-12
SLIDE 12

Operational Excellence

Science Response Time

54% 63% 81% 99% 7800 Tickets 3900 Tickets 4700 Tickets 4500 Tickets 3000 4000 5000 6000 7000 8000 9000 0% 20% 40% 60% 80% 100% Q4 2013 Q1 2014 Q2 2014 Q3 2014 Volume of tickets Percent within target

Board of Governors Meeting, September 15, 2014 32

Q2 Q3 Target: 3 Business Days

slide-13
SLIDE 13

Yellow-flagged Items – Off Target

Board of Governors Meeting, September 15, 2014 33

slide-14
SLIDE 14

Increase Information

Progress of Research Projects

Q1 Q2 Q3 Q4 Number of Projects with a Progress Report Due by Q3 2014

  • Predominantly 6-month
  • A few 12-month
  • By next Q, a few 18-month

23 57 140 Percent of Projects Meeting All Milestones Due 67% 63% 51% Average Percent of Milestones Due that Were Met 88% 87% 85%

Board of Governors Meeting, September 15, 2014 34

We are refining our approach to measuring and reporting on the progress of projects and will present additional analyses in Q4

slide-15
SLIDE 15

Increase Information

Measures of Progress of Research Projects

Board of Governors Meeting, September 15, 2014 35

67% 93% 2% 63% 76% 3% 51% 80% 4% 4% 10 20 30 40 50 60 70 80 90 100

Meeting All Milestones Meeting Recruitment Milestones Obtained IRB Approval on Schedule Payment Hold for Programmatic Reasons Contract Modification for Milestones

Percent

Q1 Q2 Q3 Q4

Coming in Q4

slide-16
SLIDE 16

Influence Research

Pipeline to Proposal Awards – Q3 Snapshot

Tier I 30 Awards

nearing completion

Pre-Engagement/ Community Building Projects

Up to $15,000/award

Tier II Partnership and Infrastructure Development Projects

Up to $25,000/award

Tier III Proposal Development Projects

Up to $50,000/award

PCORI Research Proposals

Expected in 2016

36 Board of Governors Meeting, September 15, 2014

Note: We no longer anticipate making additional awards in FY 2014 and thus will not meet our

  • riginal 2014 target of 65 projects.
slide-17
SLIDE 17

Influence Research

The Future of Pipeline to Proposal Awards

Our initial 30 awardees and first Program Office are providing valuable data and feedback for refinement of

  • ur concept and plans for future funding rounds
  • 29 out of 30 projects have met 100% of their milestones

Early in FY 2015, we plan to award:

  • 4 additional regional Program Offices, for a total of 5
  • Up to 50 additional Tier I projects
  • Up to 30 Tier II projects that have advanced from Tier I
  • Up to 50 Tier III projects

Board of Governors Meeting, September 15, 2014 37

slide-18
SLIDE 18

Influence Research

Progress of PCORnet

Status of PCORnet Major Deliverables at Q3  Common data model

  • Version 1 completed

 Governance policies

  • 2 of 8 policies in development are nearing completion

 Clinical trial

  • Topic determined

 Steering Committee and Task Force

  • Charters completed
  • Members selected and meetings occur regularly

 Communications

  • PCORnet.org launched
  • Collaborative workspace launched

Board of Governors Meeting, September 15, 2014 38

slide-19
SLIDE 19

20 40 60 80 100 Award to Contract Contracts Response Science Response Percent of Target

w/in 90 cal. days w/in 2 bus. days w/in 3 bus. days

Q1 Q2 Q3 Q4

Ambassadors Fully Trained

Q3

50 100 150 200 250 300 FY 2012 FY 2013 FY 2014 $M Targeted Broad PCORnet Pilots

10 20 30 40

Cycle I Cycle II Cycle III August 2013

Percent % of Applications Awarded % of Resubmissions Awarded

NA

Needs Attention On Target Off Target

Legend Broad Awards Success Rates Dissemination and Uptake

Board of Governors FY2014 Dashboard – Q3 (As of 6/30/2014) Our Goals: Increase Information, Speed Implementation, and Influence Research

*Influencing Research* PCORI is the catalyst for a proposal to establish the Center for Pediatric Comparative Effectiveness Research at the University of Arkansas for Medical Sciences

20 40 60 80 100 Research P2P PPRNs CDRNs Percent Meeting All

Q1 Q2 Q3 Q4

Percent of Projects Meeting All Milestones Completion of Phase I of PCORnet

Q2

Funded Research Portfolio Impact of Engagement on Merit Review 2014 Research Funding Commitment

Q3

Responsiveness

$528M Q1 Q3 100

Current Actual Target Previous Actual

Pipeline to Proposal Awards

Q4 65 1/3

  • f

Phase I Q3

2014 Expenditures on Research Awards – $M

Q1 $94M

30 X

Expense Categories External Research Commitments

20 40 60 80 100 FY 2014 FY 2015 Percent Program Program Support Admin Actual Budget 62

Q2 Q3

20 40 60 80 100

FY 2013 FY 2014 All All by $

Percent Other Screening Treatment Diagnosis Prevention

NA

Other 2014 Expenditures – $M

Q2 Q1 $89M Q3

2014 Staffing Plan – Number of People

Q1 Q2 Q3 164

NA=Not Applicable

NA NA 63 136 NA NA NA 30 191 20 40 60 80 100 Journal Articles Number 1,000 2,000 3,000 4,000 5,000 6,000 7,000 Web Views 2 4 6 8 10 Citations

Q1 Q2 Q3 Q4 (of Methodology Standards)

Priority Topics to Be Awarded in 2014

Q3 5

2

and

Q1

40 60 80 100 Cycle II Cycle III Aug 13 Winter 14

Percent of reviewers who think patient & stakeholder reviewers provide valuable input

slide-20
SLIDE 20

Appendix

Additional Background Slides to Respond to Questions

Board of Governors Meeting, September 15, 2014 40

slide-21
SLIDE 21

Influence Research

Related Evaluation Activities

Our regular survey of Merit Reviewers is one example of our evaluation activities – other examples include: To evaluate Merit Review, we also

  • Analyze scores (see recent publication by Fleurence, et al.)
  • Analyze written application critiques
  • Conduct small group discussions with merit reviewers

We also evaluate the Impact of Engagement on

  • Topic prioritization
  • All phases of the research process
  • Dissemination

To better understand our stakeholders, we also Survey and conduct focus groups with

  • LOI Submitters and Applicants
  • Patients and Caregivers
  • Clinicians and Researchers
  • Industry, Payer, and Purchaser Stakeholders

Board of Governors Meeting, September 15, 2014 41

slide-22
SLIDE 22

Influence Research

Progress of PCORnet: CDRNs at 3 months

Task  Network Organizational Metadata complete – 3/11  PopMedNet capability established (DataMartClient installation complete) – 2/11  Performed test query run on fake dataset – 0/11  Developed plans for streamlined Institutional Review Board – 5/11

 Patients currently engaged/finalizing engagement in governance – 11/11

 Developing ways to disseminate research results – 2/11  Developing plans for conducting clinical trial without disrupting healthcare

  • perations – 8/11

Meeting 100% of Milestones Due in Q3 – 5/11

Board of Governors Meeting, September 15, 2014 42

slide-23
SLIDE 23

Influence Research

Progress of PCORnet: PPRNs at 3 months

Board of Governors Meeting, September 15, 2014 43

9 COMMON CONDITION PPRNs

 Data domains:

 demographic 9/9

 vital signs 5/9  enrollment, diagnosis data, and encounter data 4/9

 Patient portals:

 Developing 9/9  Beta launch 6/9

 Launched 1/9

 IRB Approved: 4/9

 Patient Engagement: 9/9 with patients

in governance Meeting 100% of Milestones Due Q3: 8/9

9 RARE CONDITION PPRNs + CENA

 Data domains:

 demographic 9/9

 vital signs 6/9 (1/9 undecided)  enrollment, diagnosis data, and encounter data: 8/9 ((1/9 undecided)

 Patient portals: 9/9

 Launched and enrolling patients: 2/9

 IRB Approval:

 Full: 3/9  Partial: 4/9  Under review: 1/9  Not submitted yet: 1/9

 Governance Structures Developed: 9/9  Patient Engagement: 9/9 with patients in

governance Meeting 100% of Milestones Due Q3 : 6/9

slide-24
SLIDE 24

1302 1099 1038 1050 850 1135 970 482 434 557 461 355 508 76 54 79 95 25 51 71 45 34 22 13 13

200 400 600 800 1000 1200

Cycle I Cycle II Cycle III August 2013 Winter 2014 Spring 2014

Letters of Intent Letters of Intent Accepted Applications Applications that were Resubmissions Awards Awards that were Resubmissions

Increase Information

LOIs, Applications, and Awards: Broad PFAs

44 Board of Governors Meeting, September 15, 2014

slide-25
SLIDE 25

Increase Information

Point of Reference for Resubmissions

Board of Governors Meeting, September 15, 2014 45

2013 New - First Submission (A0) 33,740 3,144 9.3% $1,375,817,409 16,999 1,440 8.5% $624,473,802 2013 New with Resubmissions (A1) 10,620 3,345 31.5% $1,221,237,108 6,263 1,892 30.2% $808,914,272 2013 Continuations (A0) 3,030 873 28.8% $497,759,255 2,755 719 26.1% $310,767,843 2013 Continuations with Resubmissions (A1) 2,030 898 44.2% $409,208,222 1,876 804 42.9% $320,675,931 2013 Supplements 161 50 31.1% $9,025,718 151 47 31.1% $8,079,504 Fiscal Year Type1 RESEARCH PROJECT GRANTS3 R01-EQUIVALENT GRANTS4 and Submission Number

2

Number of Applications Reviewed Number of Applications Awarded Success Rate5 Total Funding6 Number of Applications Reviewed Number of Applications Awarded Success Rate5 Total Funding6

Data for 2013 from NIH “RePORT” – Success Rates/Research Project Grants/Spreadsheet #10

  • Percent of NIH applications that were resubmissions in 2013 = 24% and 27% (for the 2 sets shown above)
  • PCORI’s last four cycles have averaged 17% resubmissions

5% 12% 13% 10% 29% 24% 10 20 30 40

Cycle I Cycle II Cycle III August 2013

Percent % of Applications Awarded % of Resubmissions Awarded

NA NA

In Q4, we will have two additional PCORI funding cycles for this analysis

slide-26
SLIDE 26

Increase Information

Status of Targeted Topics as of Q3

Topic Status FY 2014 FY 2015

Awarded  Asthma (Q1) – 8 Projects, $23M  Falls (Q3) – 1 Project, $30M Year-to-Date Total 2 Topics, 9 Projects, $53M To Be Awarded

  • Care Transitions – 1 Project, $15M
  • Fibroids – 1-2 Projects, $20M
  • Obesity – 2 Projects, $20M
  • Hypertension – up to 2 Projects, $25M
  • 19 Pragmatic – up to 27 Projects, $270M

Posted

  • 19 on Pragmatic Studies List

Nearing SOC or Board Vote

  • Perinatal
  • Hepatitis C

Under Consideration 77 Topics Cumulative Total 2 Topics, 9 Projects, $53M Target for Year 5 Topics, ~24 Projects, $113M To be established

Board of Governors Meeting, September 15, 2014 46

slide-27
SLIDE 27

Increase Information

Funds Announced for Solicitations Compared with Funds Awarded

Board of Governors Meeting, September 15, 2014 47

120 96 108 80 17 68 72 81 20 15 90 90 41 89 114 74 23 94 55 20 40 60 80 100 120 $Millions Funding Level Announced Funding Level Awarded

slide-28
SLIDE 28

Increase Information

Funds Committed for External Research

Board of Governors Meeting, September 15, 2014 48

31 31 9 94 103 244 74 318 50 23 73 100 200 300 400 500 600 FY 2012 FY 2013 FY 2014 Q3 Cumulative Targeted PFAs Broad PFAs PCORnet Pilot Projects

$Millions

slide-29
SLIDE 29

Increase Information

Composition of Current Total Portfolio (by projects and $)

Prevention 15% Diagnosis 2% Treatment 74% Screening 6% Other 3%

Prevention Diagnosis Treatment Screening Other

N = 171

(Cycles I-III and August 2013, not including Methods, including Falls Study)

49

Projects included in “Other” overlap 2 or more categories

Board of Governors Meeting, September 15, 2014

Prevention 24% Diagnosis 2% Treatment 66% Screening 6% Other 2%

N = $315M

(Cycles I-III and August 2013, not including Methods, including Falls Study)

slide-30
SLIDE 30

Influence Research

Communications

Board of Governors Meeting, September 15, 2014 50

Communications Website E-mail Social Media

Because our early Twitter stats were low, year-to-year comparisons against targets aren’t useful. We’re tracking follower growth and impressions and working on a more sophisticated reach analysis. Our Q3 unique website visitors numbers continue to beat our target. We continue to exceed industry standards for open and click-through rates by a wide margin.

Media Coverage

We continue to grow the number of mentions of our work in general/trade media.

2012 2013 2014

Q1

  • 45

150 Q2 47 48 212 Q3 46 89 149 Q4 26 117

  • 10,000

20,000 30,000 40,000 50,000

Monthly Unique Traffic

2012 2013 2014

20,000 40,000 60,000 Q1 2014 Q2 2014 Q3 2014

Website Unique Traffic

Target Actual 0% 10% 20% 30% Q1 2014 Q2 2014 Q3 2014

Email Open Rate

Target Actual 0% 5% 10% 15% 20% Q1 2014 Q2 2014 Q3 2014

Email Click-Thru Rate

Target Actual 1,000 2,000 3,000 4,000 5,000

Monthly Twitter Follower Growth

slide-31
SLIDE 31

51

3 33 22 20 40 Q1 Q2 Q3 Q4 Number of LOIs Received

  • Letters of Intent (LOI) and

applications are currently received and reviewed on a rolling basis

  • Beginning in October 2014,

we will accept and review them on a quarterly cycle

Influence Research

Eugene Washington Engagement Awards

Board of Governors Meeting, September 15, 2014

slide-32
SLIDE 32

Influence Research

Eugene Washington Engagement Awards

Board of Governors Meeting, September 15, 2014 52

58 LOIs Received 27 Invited to Submit Application 1 Declined 11 Not Yet Received 8 Under Review 7 Awards Made 1 Under Review 30 Declined

slide-33
SLIDE 33

Influence Research

Eugene Washington Engagement Awards

53 Board of Governors Meeting, September 15, 2014

2 5

Knowledge Awards Training and Development Awards

N = 7 Awards in Total

Award Categories Not Yet Represented

  • Dissemination
  • Meeting/Conference Support
  • PCORI Pilot Projects Learning Network

Dissemination and Implementation Support