Dashboard Review Third Quarter of FY 2014 Joe Selby, MD, MPH - PowerPoint PPT Presentation

Dashboard Review Third Quarter of FY 2014 Joe Selby, MD, MPH Executive Director Board of Governors Meeting, September 15, 2014 21

First Message from the Dashboard: Notice the Colors Legend For those of us who can’t see On Target red, the red Off Target boxes would Needs show up in gray Attention scale as dotted Board of Governors Meeting, September 15, 2014 22

Legend *Influencing Research* On Target PCORI is the catalyst for a proposal to establish the Center for Pediatric Comparative Target Board of Governors Off Target NA Effectiveness Research at the University of Arkansas for Medical Sciences FY2014 Dashboard – Q3 Previous Actual Needs Attention Our Goals: Increase Information, Speed Implementation, and Influence Research (As of 6/30/2014) NA=Not Applicable X Current Actual Broad Awards Success Rates Funded Research Portfolio Dissemination and Uptake 100 40 % of Applications Awarded 100 7,000 10 Other Q1 % of Resubmissions Awarded 80 6,000 80 8 Percent Screening Q2 30 Percent 5,000 60 Number Treatment Q3 60 6 4,000 40 20 Diagnosis Q4 3,000 40 4 20 Prevention 2,000 10 0 20 2 1,000 FY 2013 FY 2014 All All by $ NA NA 0 0 0 0 Impact of Engagement on Merit Review Cycle I Cycle II Cycle III August 2013 Journal Web Views and Citations (of Methodology Standards) Articles Percent of 100 Priority Topics to Be Awarded in 2014 reviewers who Pipeline to Proposal Awards 80 think patient & stakeholder 5 65 2 60 30 reviewers provide Q3 Q1 Q3 Q4 valuable input 40 Cycle II Cycle III Aug 13 Winter 14 Ambassadors Fully Trained 2014 Research Funding Commitment Percent of Projects Meeting All Milestones 100 63 100 $528M 191 Q1 Q3 80 Percent Meeting All Q3 Q2 60 Completion of Phase I of PCORnet Q3 2014 Expenditures on Research Awards – $M 40 1/3 Q4 of 20 $94M 62 Q1 Q2 Q3 NA NA NA Phase I 0 Q1 Q2 Q3 Research P2P PPRNs CDRNs Expense Categories External Research Commitments Responsiveness Other 2014 Expenditures – $M 100 300 100 Percent of Target Q1 80 250 80 Targeted $89M 30 Program Q2 60 200 Percent Broad 60 Q1 Q2 Q3 $M 40 150 Program PCORnet Q3 40 2014 Staffing Plan – Number of People Support 20 100 Pilots 20 Q4 NA Admin 0 50 164 136 0 Award to Contracts Science 0 Contract Response Response FY 2014 FY 2015 Q1 Q2 Q3 FY 2012 FY 2013 FY 2014 Actual Budget w/in 90 cal. days w/in 2 bus. days w/in 3 bus. days

Noteworthy Items Board of Governors Meeting, September 15, 2014 24

Influence Research *Influencing Research* PCORI is the catalyst for a proposal to establish the Center for Pediatric Comparative Early Signs Effectiveness Research at the University of Arkansas for Medical Sciences Dr. Debra Fiser was a critical care physician for 14 years, then Chair of the University of Arkansas for Medical Sciences (UAMS) Department of Pediatrics for 11 years, then served as Dean of the UAMS College of Medicine for 7 years. She said as a capstone for her career, she wanted to help others in her department achieve success by mentoring young clinicians and faculty. She particularly wanted to help physicians who have not been involved in research in the past to see the relevance of research in their work and to build capacity to engage in Comparative Effectiveness Research (CER). Dr. Fiser stepped down as Dean in May 2013 and attended the PCORI Engagement Workshop in Memphis, Tennessee, in August of 2013, which provided inspiration for her proposal to establish a Center for Pediatric Comparative Effectiveness Research at UAMS. She saw the work PCORI is doing to fund patient-centered CER as the catalyst UAMS needed to invest in training faculty in CER. The objective of the Center is to recruit, train, and mentor faculty in patient-centered CER methods . It is up and running as of July 1, 2014, and currently has funding for five years to recruit five half-time faculty each with three years of support. The Center is focused on building capacity by preparing pediatric clinicians and faculty who are relatively new to research to conduct patient-centered CER , with an end goal of successfully applying to PCORI for funding. Board of Governors Meeting, September 15, 2014 25

Increase Information Merit Reviewer Survey Details We asked merit reviewers to rate their agreement with the following statement: The patient/stakeholder reviewers provided valuable input during the discussion (Strongly Disagree, Somewhat Disagree, Neither Agree nor Disagree, Somewhat Agree, and Strongly Agree) Survey Percent Who Agree Survey Sample Size Completion Rate Somewhat or Strongly Cycle II 205 94% 89% 167 97% 85% Cycle III 286 91% August 2013 93% 209 83% Winter 2014 94% Surveys of merit reviewers are conducted immediately following each in-person merit review and are live in the field for 2 weeks . 26 Board of Governors Meeting, September 15, 2014

Percent of merit reviewers who agree with the statement: “The patient/stakeholder reviewers provided valuable input during the discussion” 100% 2% 10% 19% 3% 90% 16% 6% % Somewhat 12% 17% 24% agree 80% 20% % Strongly 70% 29% agree 23% 60% 50% 96% 90% 89% 85% 82% 81% 40% 80% 75% 71% 69% 30% 56% 56% 20% 10% 0% Cycle 2 Cycle 3 August Winter Cycle 2 Cycle 3 August Winter Cycle 2 Cycle 3 August Winter 2013 2014 2013 2014 2013 2014 Patients Stakeholders Scientists Board of Governors Meeting, September 15, 2014 27

Percent of merit reviewers who agree with the statement: “ The scientific reviewers provided valuable input during the discussion” 100% 8% 6% 17% 12% 13% 15% 90% 11% 13% 18% 9% 18% 15% 80% 70% 60% % Somewhat agree 50% 90% 89% % Strongly 85% 84% 83% 84% 83% 82% 82% 40% 79% 76% agree 73% 30% 20% 10% 0% Cycle II Cycle III August Winter Cycle II Cycle III August Winter Cycle II Cycle III August Winter 2013 2014 2013 2014 2013 2014 Patients Stakeholders Scientists Board of Governors Meeting, September 15, 2014 28

Percent of merit reviewers who agree with the statement: “ Overall, scientific reviewers were receptive to input from patient and stakeholder reviewers” 100% 90% 15% 16% 17% 16% 21% 17% 80% 32% 29% 17% 18% 32% 70% 24% 60% % Somewhat 50% agree 83% 81% 40% 78% % Strongly 78% 73% 72% 69% agree 68% 64% 63% 30% 58% 51% 20% 10% 0% Cycle II Cycle III August Winter Cycle II Cycle III August Winter Cycle II Cycle III August Winter 2013 2014 2013 2014 2013 2014 Patients Stakeholders Scientists Board of Governors Meeting, September 15, 2014 29

Increase Information Our first published study of Merit Review Board of Governors Meeting, September 15, 2014 30

Influence Research PCORI Ambassadors Program Ambassadors Enrolled by Stakeholder Group (Cumulative) 30 Ambassadors Enrolled by Geographic Region 25 20 Q1 Midwest: 18 Q2 15 West: 11 Northeast: 30 Q3 10 Southwest: 15 Southeast: 26 5 0 Board of Governors Meeting, September 15, 2014 31

Operational Excellence Science Response Time Q2 Q3 99% 100% 9000 81% 7800 Tickets 8000 80% Percent within target 7000 63% Volume of tickets 60% Target: 54% 3 Business 6000 Days 40% 5000 4700 Tickets 4500 Tickets 20% 4000 3900 Tickets 0% 3000 Q4 2013 Q1 2014 Q2 2014 Q3 2014 Board of Governors Meeting, September 15, 2014 32

Yellow-flagged Items – Off Target Board of Governors Meeting, September 15, 2014 33

Increase Information Progress of Research Projects Q1 Q2 Q3 Q4 Number of Projects with a Progress Report Due by Q3 2014 • Predominantly 6-month 23 57 140 • A few 12-month • By next Q, a few 18-month Percent of Projects Meeting All 67% 63% 51% Milestones Due Average Percent of Milestones 88% 87% 85% Due that Were Met We are refining our approach to measuring and reporting on the progress of projects and will present additional analyses in Q4 Board of Governors Meeting, September 15, 2014 34

Increase Information Measures of Progress of Research Projects 100 93% Q1 Q2 Q3 Q4 90 80% 80 76% 67% 70 63% 60 Percent 51% 50 40 30 20 10 Coming in Q4 4% 4% 3% 2% 0 Meeting All Meeting Recruitment Obtained IRB Payment Hold for Contract Modification Milestones Milestones Approval on Programmatic for Milestones Schedule Reasons Board of Governors Meeting, September 15, 2014 35

Influence Research Pipeline to Proposal Awards – Q3 Snapshot Tier I Tier III Tier II 30 Awards PCORI Proposal Partnership and nearing completion Research Development Infrastructure Pre-Engagement/ Projects Proposals Development Community Projects Expected in 2016 Building Projects Up to $25,000/award Up to $50,000/award Up to $15,000/award Note: We no longer anticipate making additional awards in FY 2014 and thus will not meet our original 2014 target of 65 projects. Board of Governors Meeting, September 15, 2014 36

Influence Research The Future of Pipeline to Proposal Awards Our initial 30 awardees and first Program Office are providing valuable data and feedback for refinement of our concept and plans for future funding rounds  29 out of 30 projects have met 100% of their milestones Early in FY 2015, we plan to award:  4 additional regional Program Offices , for a total of 5  Up to 50 additional Tier I projects  Up to 30 Tier II projects that have advanced from Tier I  Up to 50 Tier III projects Board of Governors Meeting, September 15, 2014 37