Dashboard Review End of FY 2014 Joe Selby , MD, MPH Executive - - PowerPoint PPT Presentation

Dashboard Review

Board of Governors Meeting, December 8, 2014 1

End of FY 2014

Joe Selby , MD, MPH Executive Director

* As presented during the PCORI Board of Governors meeting on December 8, 2014

100 80 60 40 20 Award to Contracts Science Contract HelpDesk HelpDesk

w/in 90 days 100% Customer Satisfaction

Percent of Target Q1 Q2 Q3 Q4

Ambassadors to Be Fully Trained = 100

300 250 200 150 100 50 350 FY 2012 FY 2013 FY 2014 $M Targeted Broad PCORnet Pilots

10 20 30 40

Cycle I Cycle II Cycle III August Winter 2013 2014

Priority Topics to Be Awarded = 5

Spring 2014

Percent % of Applications Awarded 100 % of Applications that were Resubmissions % of Resubmissions Awarded Needs Attention On Target Off Target

Legend Success Rates for Broad Awards Composition of Funded Research Portfolio Dissemination and Uptake Indicators

Board of Governors FY2014 Dashboard – Q4 (As of 9/30/2014) Our Goals: Increase Information, Speed Implementation, and Influence Research

*Influencing Research* Working with PCORI influenced Geisinger Health System to incorporate patient engagement into its strategic plan for research and require it for all projects

100 80 60 40 20 Percent Meeting All

Responsiveness

Q4

Expense Categories External Research Commitments

80 60 40 20 100 FY 2014 FY 2015 Actual Budget Percent Program Program Support Admin

Q4

Percent

NA

Other Expenditures Planned = $89M

Research P2P PPRNs CDRNs

Completion of Research Projects

Expected Actual Q1 Q2 Q3 Q4

Percent of Projects Meeting All Milestones Portion of PCORnet I to Be Completed = 1/3 Funds to Be Committed to Research = $528M

$528M 100

NA=Not Applicable X Current Actual Target Previous Actual

Pipeline to Proposal Awards to Be Made = 65

30

Q4 65 1/3

• f

Phase I

Funds to Be Expended on Research Awards=$94M

$94M

80 60 40 20

FY 2013 FY 2014 All All by $ Other Treatment Diagnosis Prevention 45

Q4 $89M

Planned Staffing Level = 164 People

153

Q4 164

NA NA NA NA NA 10 20 30 40 50 Number 7,000 6,000 5,000 4,000 3,000 2,000 1,000 Web Views and Citations

(of Methodology Standards)

5 10 15 20 25

Q1 Q2 Q3 Q4

2 4

Q1 Q2 Q3 Q4

Number of Projects

113

5

5 Q4

350

Q4

82

Q4

Journal Articles by Awardees

New and Noteworthy Items

100 80 60 40 20 Award to Contracts Science Contract HelpDesk HelpDesk

w/in 90 days 100% Customer Satisfaction

Percent of Target Q1 Q2 Q3 Q4

Ambassadors to Be Fully Trained = 100

300 250 200 150 100 50 350 FY 2012 FY 2013 FY 2014 $M Targeted Broad PCORnet Pilots 10 20 30 40 NA NA Cycle I Cycle II Cycle III August Winter Spring 2013 2014 2014 Percent % of Applications Awarded % of Applications that were Resubmissions % of Resubmissions Awarded Needs A en on On Target Off Target

Legend

Target

Success Rates for Broad Awards Composi on of Funded Research Por olio Dissemina on and Uptake Indicators

Board of Governors FY2014 Dashboard – Q4 (As of 9/30/2014) Our Goals: Increase Informa on, Speed Implementa on, and Influence Research

*Influencing Research* Working with PCORI influenced Geisinger Health System to incorporate pa ent engagement into its strategic plan for research and require it for all projects

100 80 60 40 20 Research P2P PPRNs CDRNs

Comple on of Research Projects

Expected Actual Percent Meeting All Q1 Q2 Q3 Q4

Percent of Projects Mee ng All Milestones Por on of PCORnet I to Be Completed = 1/3 Funds to Be Commi ed to Research = $528M Responsiveness $528M Q4 100

NA=Not Applicable X Current Actual Previous Actual

Pipeline to Proposal Awards to Be Made = 65

30

Q4 65 1/3

• f

Phase I Funds to Be Expended on Research Awards=$94M

113

$94M Expense Categories External Research Commitments

100 80 60 40 20 FY 2014 FY 2015 Actual Budget Percent Program Program Support Admin

Q4

100 80 60 40 20 FY 2013 FY 2014 All All by $ Percent Other Treatment Diagnosis Prevention

NA

Other Expenditures Planned = $89M

45

Q4 $89M Planned Staffing Level = 164 People Q4 164

NA NA NA 10 20 30 40 50 Number 7,000 6,000 5,000 4,000 3,000 2,000 1,000 Web Views and Citations (of Methodology Standards) 5 10 15 20 25 Q1 Q2 Q3 Q4

Priority Topics to Be Awarded = 5

2 4

Q1 Q2 Q3 Q4 Number of Projects 5 5

Q4

350

Q4

153 82

Q4

Journal Ar cles by Awardees

Board of Governors Meeting, December 8, 2014 3

Goal 3: Influence Research

Early Signs

Geisinger Health System (GHS) leadership met to evaluate their progress toward achieving goals in their 10 year research strategic plan. They discussed changes which had occurred in the research environment over those 6 years, to which they needed to adapt. They identified patient engagement in research as key area of emphasis.

• GHS had received 2 PCORI awards over that period of time

Patient Engagement Work Group was formed to explore the place of patient engagement in GHS research going forward

• Key recommendation is that patient engagement be the default at GHS, not

the exception

• Work Group’s goal is to create an atmosphere conducive to engaging patients in

GHS research by raising awareness and making resources available to GHS research community

• Several PCORI engagement concepts were central to the work group’s

thinking and subsequently appeared in their final report and recommendation to leadership

Board of Governors Meeting, December 8, 2014 4

*Influencing Research* Working with PCORI influenced Geisinger Health System to incorporate patient engagement into its strategic plan for research and require it for all projects

GHS Patient Engagement Framework “..as we navigate our way toward enhancing patient engagement in

research and discovery at Geisinger, PCORI has been and will continue to be both guide and anchor for that effort”*

GHS attributes their new focus on patient engagement in research to:

Board of Governors Meeting, December 8, 2014 5

• Internal culture that has always valued a

patient centered approach (readily adaptable) External forces, such as PCORI, which have advanced the national conversation around patient engagement and provided invaluable resources (both financial and intellectual) for making it possible

• PCORI resources have served as a “lead

impetus,”* and heavily informed GHS’s conceptualization of their own patient engagement framework Patient Engagement Goals have been built into GHS Strategic Plan and personnel goals of GHS leadership

*Dan Davis, PhD, Director, GHS Bioethics and Patient Engagement Work Group Lead

Number of Journal Articles Concerning PCORI

5 3 3 8 11 23 22 12 56 10 20 30 40 50 60 Articles by Awardees Articles by or about PCORI Articles that Cite or Mention PCORI Work Number of Articles Journal Articles Published by Quarter

Q1 Q2 Q3 Q4

Goal 2: Speed Implementation

20 Board of Governors Meeting, December 8, 2014

Expected Number of Contracts Ending per Quarter

(Includes estimates for some contract modifications in process)

2 1 30 1 1 16 1 1 1 13 40 35 32 56 15 13 21 20 5 6 2 11 5 8 3 4 10 30 50 60 Q1 2014 Q2 Q3 2014 Q4 Q1 2015 Q2 Q3 2015 Q4 Q1 Q2 2016 Q3 2016 Q4 Q1 2017 Q2 Q3 2017 Q4 Q1 2018 Number of Projects Fiscal Year

Pilot CER Methods

Goal 1: Increase Information

21 Board of Governors Meeting, December 8, 2014

Contracts’Help Desk Response Time

96% 99% 99% 99% 3200 Tickets 3200 Tickets 2100 Tickets 1700 Tickets 1,000 20% 500 1,500 2,000 80% 2,500 3,000 3,500 0% 40% 60% 100% Q1 Q2 Q3 Q4 Volume of tickets Percent within target T arget: 3 Business Days

Goal 1: Increase Information

22 Board of Governors Meeting, December 8, 2014

Satisfaction with Contracts Help Desk Response

Data from software that handles help desk tickets

Staff follow-up with all respondents who indicated that they were dissatisfied

93% 93% 93% 94% 699 (22%) Rated 609 (19%) Rated 335 (16%) Rated 284 (17%) Rated 200 20% 100 300 400 500 600 700 100% 800 0% 40% 60% 80% Q1 Q2

Industry average = 85% of respondents satisfied

Q3 Q4 Number (percent) of tickets rated Percent satisfied with response

Goal 1: Increase Information

Board of Governors Meeting, December 8, 2014 9

Science Help Desk Response Time

63% 81% 99% 99% 292 Tickets 599 Tickets 273 Tickets 384 Tickets 200 20% 100 300 400 500 600 700 0% 40% 60% 80% 100% Q1 Q2 Q3 Q4 Volume of tickets Percent within target T arget: 3 Business Days

Goal 1: Increase Information

Board of Governors Meeting, December 8, 2014 10

Satisfaction with Science Help Desk Response

Data from software that handles help desk tickets

Staff follow-up with all respondents who indicate that they were dissatisfied

100% 100 85% 90% 90% 91% 40 (14%) Rated 70 (12%) Rated 53 (19%) Rated 58 (15%) Rated 20 40 60 80 0% 20% 40% 60% 80% Q1 Q2

Industry average = 85% of respondents satisfied

Q3 Q4 Number (percent) of tickets rated Percent satisfied with response

Goal 1: Increase Information

Board of Governors Meeting, December 8, 2014 11

Applicant Satisfaction with Support

Information about PCORI’s Applicant Survey

Our Applicant Survey asks about the following sources of support for applicants: pfa@pcori.org – “Contracts” sciencequestions@pcori.org – “Science” Science Program T elephone Help Line

Goal 1: Increase Information

Board of Governors Meeting, December 8, 2014 12

Completion Rates of Applicant Surveys

The survey is administered upon submission of application and is anonymous

Cycle August 2013 Winter 2014 Spring 2014 Completion Rate 27% 61% 43% Number of Respondents 626 355 519

Ratings of Helpfulness of Email Support by Cycle

Data from PCORI’s Applicant Survey (Contracts & Science combined)

16% 13% 14% 9% 5% 8% 8% 8% 7% 27% 18% 33% 36% 56% 42%

122 Responses

20 40 60 80 100 120 140 160 0% 20% 40% 60% 80% 100% August 2013 Winter 2014 Spring 2014 Number of Responses Percent Agreeing or Disagreeing with Statement

Survey Item: The answer I received by email was helpful

Survey Item: The email response I received answered my question

128 Responses 151 Responses

Goal 1: Increase Information

Board of Governors Meeting, December 8, 2014 13

Strongly Disagree Somewhat Disagree Neither Agree nor Disagree Somewhat Agree Strongly Agree Number of respondents

Ratings of Helpfulness of Email Support from Contracts Compared with that from Science for Spring Cycle (only)

(Note small number of respondents for Science)

9% 17% 9% 0% 7% 17% 32% 42% 43% 25% 0% 20% 40% 60% 100%

Survey Item: The email response(s) I received answered my question(s)

80% pfa@pcori.org (N=151, 66%) sciencequestions@pcori.org (N=12, 5%) Spring 2014 Percent Agreeing or Disagreeing with Statement Strongly Disagree Somewhat Disagree Neither Agree nor Disagree Somewhat Agree Strongly Agree

Goal 1: Increase Information

Board of Governors Meeting, December 8, 2014 14

Ratings of Helpfulness of Telephone Support to Applicants Requesting a Phone Call by Cycle

Data from PCORI’s Applicant Survey

Survey Item: The answer(s) I received during the telephone call with a

43% 11% 10% 0% 4% 4% 4% 9% 5% 22% 22% 22% 26% 59% 59%

23 responses 46 responses 58 responses

10 20 30 40 50 60 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% August 2013 Strongly Disagree or No Call Occurred Neither Agree nor Disagree Strongly Agree Winter 2014 Spring 2014 Somewhat Disagree Somewhat Agree Number who requested a phone call Number who responded that they requested a call Percent Agreeing or Disagreeing with Statement

PCORI representative was helpful Goal 1: Increase Information

Board of Governors Meeting, December 8, 2014 15

Yellow-Flagged Items – More than 10% Off Target

100 80 60 40 20 Award to Contracts Science Contract HelpDesk HelpDesk

w/in 90 days 100% Customer Satisfaction

Percent of Target Q1 Q2 Q3 Q4

Ambassadors to Be Fully Trained = 100

300 250 200 150 100 50 350 FY 2012 FY 2013 FY 2014 $M Targeted Broad PCORnet Pilots 10 20 30 40 NA NA Cycle I Cycle II Cycle III August Winter Spring 2013 2014 2014 Percent % of Applications Awarded % of Applications that were Resubmissions % of Resubmissions Awarded

Needs A en on On Target Off Target

Legend

Target

Success Rates for Broad Awards Composi on of Funded Research Por olio Dissemina on and Uptake Indicators Board of Governors FY2014 Dashboard – Q4 (As of 9/30/2014) Our Goals: Increase Informa on, Speed Implementa on, and Influence Research *Influencing Research* Working with PCORI influenced Geisinger Health System to incorporate pa ent engagement into its strategic plan for research and require it for all projects

100 80 60 40 20 Research P2P PPRNs CDRNs

Comple on of Research Projects

Expected Actual Percent Meeting All Q1 Q2 Q3 Q4

Percent of Projects Mee ng All Milestones Por on of PCORnet I to Be Completed = 1/3 Funds to Be Commi ed to Research = $528M Responsiveness

$528M Q4 100

NA=Not Applicable X Current Actual Previous Actual

Pipeline to Proposal Awards to Be Made = 65

30

Q4 65 1/3

• f

Phase I Funds to Be Expended on Research Awards=$94M

113

$94M

Expense Categories External Research Commitments

100 80 60 40 20 FY 2014 FY 2015 Actual Budget Percent Program Program Support Admin

Q4

100 80 60 40 20 FY 2013 FY 2014 All All by $ Percent Other Treatment Diagnosis Prevention

NA

Other Expenditures Planned = $89M

45

Q4 $89M

Planned Staffing Level = 164 People

Q4 164

NA NA NA 10 20 30 40 50 Number 7,000 6,000 5,000 4,000 3,000 2,000 1,000 Web Views and Citations

(of Methodology Standards)

5 10 15 20 25 Q1 Q2 Q3 Q4

Priority Topics to Be Awarded = 5

2 4 Q1 Q2 Q3 Q4

Number of Projects 5 5

Q4

350

Q4

153 82

Q4

Journal Ar cles by Awardees

30 Board of Governors Meeting, December 8, 2014

Commitments and Expenditures

31 Board of Governors Meeting, December 8, 2014

Progress of Research Projects – Milestones

We continue to refine our approach to measuring and reporting

• n the progress of projects and will present additional analyses

as we have sufficient data for them Goal 1: Increase Information

18

Q1 Q2 Q3 Q4 Number of Projects with a Progress Report Due

• Predominantly 6-month report only
• 80 with 12-month report also
• A few with18-month report due next Q

23 57 140 186 Percent of Projects Meeting All Milestones Due 67% 63% 51% 45% Average Percent of Milestones Due that Were Met 88% 87% 85% 84% Percent of Projects Meeting Recruitment Milestones (47 projects with a recruitment milestone due by Q4 2014) NA NA NA 57%

Board of Governors Meeting, December 8, 2014

Additional Measures of Progress of Projects

67% 93% 2% 3%

Terminated

63% 76% 51% 80% 4% 4%4% 45% 57% 91% 1% 10 40 30 20 50 60 70 80 90 100

Meeting All Milestones Meeting Recruitment Milestones Obtained IRB Approval on Schedule Payment Hold for Programmatic Reasons Contract Modification for Milestones

Percent

Q1 Q2 Q3 Q4

NA

Goal 1: Increase Information

19 Board of Governors Meeting, December 8, 2014

Discussion and Questions

Are there new metrics suggested by today’s presentation and discussion? Are there metrics that should be retired?

20 Board of Governors Meeting, December 8, 2014

100 80 60 40 20 Award to Contracts Science Contract HelpDesk HelpDesk

w/in 90 days 100% Customer Satisfaction

Percent of Target Q1 Q2 Q3 Q4

Ambassadors to Be Fully Trained = 100

300 250 200 150 100 50 350 FY 2012 FY 2013 FY 2014 $M Targeted Broad PCORnet Pilots

10 20 30 40 Percent

Cycle I Cycle II Cycle III August Winter 2013 2014

Priority Topics to Be Awarded = 5

Spring 2014

% of Applications Awarded 100 % of Applications that were Resubmissions % of Resubmissions Awarded

NA

Target Needs Attention On Target Off Target

Legend Success Rates for Broad Awards Composition of Funded Research Portfolio Dissemination and Uptake Indicators

Board of Governors FY2014 Dashboard – Q4 (As of 9/30/2014) Our Goals: Increase Information, Speed Implementation, and Influence Research

*Influencing Research* Working with PCORI influenced Geisinger Health System to incorporate patient engagement into its strategic plan for research and require it for all projects

100 80 60 40 20 Research P2P PPRNs CDRNs

Completion of Research Projects

Expected Actual

Percent Meeting All

Q1 Q2 Q3 Q4

Percent of Projects Meeting All Milestones Portion of PCORnet I to Be Completed = 1/3 Funds to Be Committed to Research = $528M Responsiveness

$528M Q4 100

NA=Not ApplicablX e Current Actual Previous Actual

Pipeline to Proposal Awards to Be Made = 65

30

Q4 65 1/3

• f

Phase I

Funds to Be Expended on Research Awards=$94M

113 $94M

Expense Categories External Research Commitments

80 60 40 20 100 FY 2014 FY 2015 Actual Budget Percent Program Program Support Admin

Q4

80 60 40 20

FY 2013 FY 2014 All All by $

Percent Other Treatment Diagnosis Prevention

NA

Other Expenditures Planned = $89M

45

Q4 $89M

Planned Staffing Level = 164 People

153

Q4 164

NA NA NA NA NA 10 20 30 40 50 Number 7,000 6,000 5,000 4,000 3,000 2,000 1,000 Web Views and Citations

(of Methodology Standards)

5 10 15 20 25

Q1 Q2 Q3 Q4

2 4

Q1 Q2 Q3 Q4

Number of Projects 5

5 Q4

350

Q4

82

Q4

Journal Articles by Awardees

Appendix

22

Additional Background Slides to Respond to Questions

Board of Governors Meeting, December 8, 2014

1302 1099 1038 1050 850 1135 970 482 434 557 461 355 508 7671 5445 79 95 25 51 3313 43 12 22 13

200 400 600 800 Applications that were Resubmissions Awards 1000 1200 Cycle I Cycle II Cycle III August 2013 Winter 2014 Spring 2014 Letters of Intent Letters of Intent Accepted Applications Awards that were Resubmissions

Number of LOIs, Applications, and Awards for Broad PFAs by Cycle

Goal 1: Increase Information

23 Board of Governors Meeting, December 8, 2014

Status of T argeted Topics as of Q4 2014

Goal 1: Increase Information

24

Topic Status FY 2014 FY 2015

Awarded  Asthma (Q1): 8 Projects, $23M  Falls (Q3): 1 Project, $30M  Care Transitions (Q4): 1 Project, $15M  Fibroids (Q4): 1 Project, $20M  Obesity (Q4): 2 Projects, $20M Year-to-Date Total 5 Topics, 13 Projects, $108M To Be Awarded

• Hypertension: up to 2 Projects, $25M
• 19 Pragmatic: up to 27 Projects, $270M

Posted

• 19 on Pragmatic Studies List
• 19 on Pragmatic Studies List

PFA in Development Nearing SOC and/or Board Vote

• Perinatal
• Hepatitis C

Under Consideration 95 Topics 95 Topics Cumulative Total 5 Topics, 13 Projects, $108M Target for Year 5 Topics, ~14 Projects, $108M Up to 20 Topics, 27 Projects, $337M

Board of Governors Meeting, December 8, 2014

Funds Announced for Solicitations Compared with Funds Awarded

120 96 108 80 17 68 72 81 20 20 15 15 90 90 41 40 89 114 74 23 94 55 67 20 60 80 100 120 $Millions Funding Level Announced Funding Level Awarded

Goal 1: Increase Information

25 Board of Governors Meeting, December 8, 2014

Funds Committed for External Research

31 31 9 96 105 244 196 440 50 58 108 100 200 300 400 Broad PFAs 500 PCORnet 600 700 FY 2012 FY 2013 FY 2014 Cumulative Targeted PFAs Pilot Projects

$Millions

Goal 1: Increase Information

26 Board of Governors Meeting, December 8, 2014

Composition of Current CER Portfolio (by number of projects and amount of funding)

Prevention 10% Diagnosis 3% Treatment 77% Other 10%

Prevention Diagnosis Treatment

“Other” projects include screening, systems, and those overlapping categories

Other

223 Studies in T

• tal

(All funded cycles to date, including targeted, not including Methods)

Prevention 15% Diagnosis 2% Treatment 72% Other 11%

$492M in T

• tal

(All funded cycles to date, including targeted, not including Methods)

Goal 1: Increase Information

27 Board of Governors Meeting, December 8, 2014

Progress of PCORnet at 6 months – Overall

Status of PCORnet Major Deliverables at Q4

 Common data model

• Version 2 drafted and sent out for comment
• Proposed data domains: Prescriptions Dispensed, Patient-Reported Outcomes Common Measures, Lab Results, Conditions

 Governance policies – 13 in development (up from 8 last quarter)

• Two draft policies are at step 8 of 10 total steps (Introduction, Decision-making and Leadership)
• Four draft policies are at step 7 (Distributed Research Networks, Privacy and Confidentiality, Data Security, Confidential

Business Information)

 Clinical trial

• Issued a limited PCORI Funding Announcement (PFA) soliciting a multi-network research plan for Aspirin Trial
• Principles of PCORnet Clinical Trials has been drafted and circulated to PCORI and PCORnet PIs for comment
• Clinical Trials Transformative Imitative (CTTI) Quality by Design Document is drafted and out for comment

 Steering Committee and T ask Force – meetings occur regularly  Communications

• IOM-PCORI Workshop Summary Published on Integrating Research and Practice: Health System Leaders Working Toward

High-Value Care

• JAMIA Theme Issue published on PCORnet, including 11 articles on CDRNs and 1 PPRN summary article
• Health Affairs Big Data theme issue published, including 5 PCORnet-related articles
• PCORnet staff presentations at major conferences (Health Datapalooza, Association of Health Care Journalists)

Goal 3: Influence Research

28 Board of Governors Meeting, December 8, 2014

Progress of PCORnet: CDRNs at 6 months

T ask Number of CDRNs Completing T ask

Meeting 100% of Milestones Due in Q4 – 2/11 Average Percent of Milestones Met – 85%

*ETL/ADD = Extract, Transform and Load / Annotated Data Dictionary

Goal 3: Influence Research

29

 PopMedNet capability established (DataMartClient Install Complete) 11/11  Significant disease-specific collaborations across 2 or more CDRNs 2/11  Submission of ETL/ADD form* for review by DSSNI operations team 5/11  Performed initial, simple query on real data 1/11  Developed plans for streamlined Institutional Review Board 11/11  Patients currently engaged in leadership and/or advisory roles 11/11  Plans for conducting clinical trials without disrupting healthcare operations 11/11  Exploring approaches to complete data 11/11

Board of Governors Meeting, December 8, 2014

Progress of PCORnet: PPRNs at 6 months

9 COMMON CONDITION PPRNs

 Data domains:  demographic 9/9

 vital signs 5/9  enrollment, diagnosis data, and encounter data 4/9

 Patient portals:  Developing 9/9

 Beta launch 6/9  Launched 6/9

 IRB Approval: 8/9  Patient Engagement: patients in

governance 9/9 Meeting 100% of Milestones Due Q4: 2/9

8 RARE CONDITION PPRNs + CENA

 Data domains:

 Demographic 5/9  vital signs 6/9  enrollment, diagnosis data, and encounter data: 8/9

 Patient portals launched and enrolling patients:

 PPRNs 5/8  CENA organizations 5/9

 IRB Approval:

 Full: 6/9  Partial: 3/9

 Governance Structures Developed: 9/9  Patient Engagement: patients in governance 9/9

Meeting 100% of Milestones Due Q4 : 4/9

Goal 3: Influence Research

44

Average Percent of Milestones Met – All PPRNs – 92%

Board of Governors Meeting, December 8, 2014

Communications: Website, E-mail, Social Media, and Media Coverage Metrics

Communications Website E-mail Social Media

Because our early Twitter stats were low, year-to-year comparisons against targets aren’t useful. We’re tracking follower growth and impressions and working on a more sophisticated reach analysis.

Monthly Twitter Follower Growth

Our Q4 unique website visitors numbers continue to beat our target. We continue to exceed industry standards for open and click-through rates by a wide margin.

Media Coverage

We continue to grow the number of mentions of our work in general/trade media.

20,000 40,000 60,000 Q1 Q2 Q3 Q4 2014 2014 2014 2014

Website Unique Traffic

Target Actual 6,000 5,000 4,000 3,000 2,000 1,000 0% 10% 20% 30% Q1 Q2 Q3 Q4 2014 2014 2014 2014 Target

Email Open Rate

Q1 Q2 2014 2014 Actual 20% 15% 10% 5% 0% Q3 Q4 2014 2014

Email Click-Thru Rate

60,000 50,000 40,000 30,000 20,000 10,000

Monthly Unique Traffic

2012 2013 2014

Goal 3: Influence Research

31

2012 2013 2014

Q1

• 45

150 Q2 47 48 212 Q3 46 89 149 Q4 26 117 166

Board of Governors Meeting, December 8, 2014

Adherence of Awarded CER* Applications to PCORI’s Methodology Standards at Time of Award

Adherence by Standard Category (average across 3 cycles – 101 applications) Adherence by Cycle (average across all standards) *Does not include Methods Studies

Goal 3: Influence Research

32

Standard Category % Adherence

Research Questions 77% Patient-Centeredness 96% Data Integrity/ Rigorous Analyses 82% Missing Data 100% Heterogeneity of Treatment Effects 80% Data Registries 67% Data Networks n/a Causal Inference 95% Adaptive/Bayesian Trials n/a Diagnostic Tests 80% Systematic Reviews 0%

August 2013 53 Applications Winter 2014 23 Applications Spring 2014 25 Applications

89% 92% 87%

Board of Governors Meeting, December 8, 2014