Cycle 2 2015: Improving Methods for Conducting PCOR Applicant Town - - PowerPoint PPT Presentation

Applicant Town Hall September 17, 2015

Cycle 2 2015: Improving Methods for Conducting PCOR

Agenda

Programmatic Overview Administrative Overview Merit Review Criteria Questions and Answers

Submit questions via the chat function in Meeting Bridge. Ask a question via phone (an

• perator will standby to take

your questions).

PCORI CER Methods Program and Contracts Staff

Rachael Fleurence, PhD Program Director, Methods & Infrastructure Kathryn Smith, MSc Program Assistant, Methods Ashton Burton Contracts Specialist, Pre-Award Contracts Jason Gerson, PhD Associate Director, Methods Emily Evans, PhD, MPH Program Officer, Methods Maryan Zirkle, MD, MS, MA Program Officer, Methods & Infrastructure Andrea Heckert, PhD, MPH Program Officer, Methods Kate McQueston, MPH Program Associate, Methods

Programmatic Overview

• Jason Gerson, PhD,

Associate Director, Methods

• PCORI was created to support research that provides

high-integrity, evidence-based information to patients, clinicians, and the broader healthcare community.

PCORI’s Mission

• Rigorous research methods are needed to produce

relevant, trustworthy findings that can improve patients’ healthcare outcomes.

Why Methods Matter

• Methods include systematic processes, designs, tools, and

techniques used to generate the evidence needed to answer questions about which healthcare options work best for particular patients.

Methods for PCOR Improving PCOR Methods

• Identify methodological gaps relevant to

the conduct of PCOR

Identify Methods Gaps

• Fund high impact studies which address

gaps in methodological research

Fund Research

• Disseminate and facilitate the adoption of

new methods to improve the conduct of PCOR

Disseminate Promising/Best Practices

Improving PCOR Methods: Program Goals

Improving Methods for Conducting PCOR Cycle 2 2015 PFA

pcori.org/funding/opportunities

Research Areas of Interest

• 1. Methods for Patient and Stakeholder Engagement
• 2. Methods for Patient-Centered Outcomes and Methods for

Patient-Reported Outcomes

• 3. Research Related to Human Subjects Protections
• 4. Methods to Improve Study Design
• 5. Methods to Improve Validity and Efficiency of Analyses
• 6. Methods to Support Data Research Networks

Cycle 2 2015 PFA: Improving Methods for Conducting PCOR

General Guidance from the Methods Program

• Significance
• Identify and explain the specific anticipated contributions and applications of

the proposed project to patient-centered outcomes research.

• Programmatic Overlap
• Differentiate the proposed research from projects already funded by the

PCORI Methods Program, as applicable.

• Study Design or Approach
• Provide a detailed description of the methodological work that is planned

and the specific ways in which it addresses the identified methodological gaps and the specific aims of your proposed project.

• Engagement Plans
• Review the instructions provided in the Methods Applications Guidelines

(pp. 14-15). Please see PCORI’s Engagement Rubric for additional information.

• Replication and Reproduction of Research Findings
• Review the instructions provided in the Methods Applications Guidelines

(pp. 17-18).

• Milestones
• Significant events, deliverables, tasks, and/or outcomes that
• ccur over the course of the project that mark progress

toward the project’s overall aims

• Deliverables
• Measurable and verifiable outcomes or products that a

project team must create and deliver according to the contract terms

See Appendix 1 of the Application Guidelines for examples of milestones.

PCORI Monitors Projects Against Milestones and Deliverables

Milestone Schedule

• Entered into the PCORI Online system
• Distinct from the Project Milestones and Timeline for Completion in the

Research Plan, under Research Strategy

• Must provide 3-10 milestones per six-month period
• See examples in Appendix A of the Cycle 2 Methods Application Guidelines
• http://www.pcori.org/sites/default/files/PCORI-PFA-2015-Cycle-2-Methods-

Application-Guidelines.pdf

Milestone Name Description Projected Completion Date A Contract Start Date

• 1/1/2015

B1 Patient Advisory Panel Complete selection and recruitment of patient advisory panel members 2/1/2015 B2 1st Patient Advisory Panel Meeting Convene 1st patient advisory panel meeting. 3/1/2015 B3 Phase 1 Instrument Finalize focus group instrument for Phase 1. 3/15/2015 B4 Phase 1 IRB Approval Develop IRB documentation for Phase 1. Submit and receive IRB approval for study protocol and documents. 4/1/2015 B5 Phase 1 Data Collection Complete participant recruitment for Phase 1 focus groups. Complete focus group data collection. 6/15/2015 B Report Submission Submit Progress Report, Using Interim Progress Report Template 7/1/2015 C1 Phase 1 Data Analysis Finish Phase 1 focus group data analysis. 7/15/2015 C2 2nd Patient Advisory Panel Meeting Convene 2nd patient advisory panel meeting. 8/1/2015 C3 Phase 2 Instruments Finalize focus group and in-depth interview instruments for Phase 2. 9/1/2015 C4 Phase 2 IRB Approval Develop study documentation for Phase 2. Submit and receive IRB approval for amended study protocol and study documents. 9/15/2015 C5 Phase 2 Recruitment Initiate participant recruitment for Phase 2 focus groups and in- depth interviews. 10/15/2015 C Report Submission Submit Progress Report, Using Interim Progress Report Template 1/1/2016 D1 Phase 2 Data Collection Finish Phase 2 focus groups and in-depth interviews data collection. 2/1/2016 Use action verbs that detail what will be done to accomplish the milestone Be realistic, given the resources and scope of the milestone

Example Milestone Schedule

Administrative Overview

• Ashton Burton,

Contracts Specialist, Contracts

Letter of Intent (LOI) and Application

• Full applications invited based on the information provided in

the LOI

• Changes after the LOI require PCORI approval:
• Changes to the PI
• Changes to the Institution
• Changes to the Study Design
• Changes in Engagement

PCORI Online: Application

► PI and Contact Information ► Project Information ► Key Personnel ► Milestones ► Templates and Uploads ► Save

Public Abstract

Project Information

Technical Abstract Project Narratives

Key Personnel

► PI and Contact Information ► Project Information ► Key Personnel ► Milestones ► Templates and Uploads ► Save and Review

Milestones

► PI and Contact Information ► Project Information ► Key Personnel ► Milestones ► Templates and Uploads ► Save and Review

Templates and Uploads

► PI and Contact Information ► Project Information ► Key Personnel ► Milestones ► Templates and Uploads ► Save and Review

►

Research Strategy

►

Dissemination & Implementation Potential

►

Replication & Reproducibility of Research and Data Sharing

►

Protection of Human Subjects

►

Consortium Contractual Arrangements

►

References Cited

►

Appendix

Research Plan Template

Research Strategy

• Maximum 20 pages in length
• Use the Research Plan Template as your guide
• Background
• Significance
• Study Design/Approach
• Project Milestones/Timeline
• Patient Population (if applicable)
• Recruitment Plan (if applicable)
• Estimated Racial/Ethnic and Gender Enrollment Table (if applicable)
• Research Team and Environment
• Engagement Plan
• Provide all the information requested, as outlined

in the template. Page Limit 20

Research Strategy

• Two items to note:
• Research questions and specific aims MUST be the

same as in the approved LOI unless the applicant has explicit and documented approval from the program to which they will submit.

• When completing the study design/approach section,

applicants should cite the relevant PCORI Methodology Standards.

• http://www.pcori.org/assets/2013/11/PCORI-Methodology-Report.pdf

Dissemination & Implementation

• Describe the potential for disseminating and implementing

the results of this research in other settings.

• Describe possible barriers to disseminating and

implementing the results of this research in other settings.

• Describe how you will make study results available to study

participants after you complete your analyses.

Page Limit 2

Replication & Reproducibility of Research and Data Sharing

• Describe the ability to reproduce potentially important findings

from this research in other data sets and populations.

• Describe how you will make a complete, cleaned, de-

identified copy of the final data set used in conducting the final analyses available within 90 days of the end of the final year of funding, or your data-sharing plan, including the method by which you will make this data set available, if requested.

• Propose a budget to cover costs of your data-sharing plan, if

requested (these cost do not need to be included in the Budget

Template).

Page Limit 2

• Describe the protection of human subjects who will be involved

in your project.

• Adhere to the Federal Policy for the Protection of Human

Subjects (45 CFR part 46), including the Common Rule.

• Ensure appropriate informed consent.
• Establish procedures to minimize risks to participants.
• Establish procedures to protect privacy and maintain

confidentiality.

• Ensure that key personnel are educated on human subjects

protections.

• Provide a Data and Safety Monitoring Plan that
• perates under the auspices of your institution,

as applicable. Page Limit 5

Protection of Human Subjects

• Describe the proposed components of the research project that

will be performed by subcontracted organizations.

• Explain the strengths that these partners bring to the overall

project. Page Limit 5

Consortium Contractual Arrangement

References Cited

• List the source material cited in the Research Plan
• Follow scholarly citation practice and the American Medical

Association (AMA) citation style guide Page Limit 10

Appendix

• Applicants can include additional materials that they believe

are useful (e.g., survey instruments, papers and publications from members of the research team).

• Reviewers are not required to review the appendix materials

in evaluating the application. Page Limit 10

People and Places Template - Biosketch

• Required for all key personnel
• Use NIH biosketch or PCORI’s format
• List all partners within the Key Personnel section
• Patient and/or stakeholder biosketches

Page Limit 5 Per person

People and Places Template – Project / Performance Site(s)

• Demonstrate that the proposed facilities have the

appropriate resources required to conduct the project to plan, within budget, and on time.

• Provide a description of the facilities that will be used

during the project, including capacity, capability, characteristics, proximity, and availability to the

project.

Page Limit

15

Professional Profile/Biosketch

Letters of Support

• Letters of support should be addressed to the PI to demonstrate

the commitment of key personnel and supporting organizations to the proposed project.

• Letters of support should clearly reflect the substantive

involvement and material contribution to be provided by the signatory parties, and are meant to substantiate the commitment

• f collaboration of all forms.
• Letters of support should be organized in the following manner:
• Letters of organizational support
• Letters of collaboration
• Letters confirming access to patient populations, data sets, and additional

resources

Budget Templates Three budget sections must be submitted as part of the online application process:

Detailed Budget Budget Summary Budget Justification NOTE

A detailed budget is needed for each year of the

• project. Complete each budget section for the prime

applicant and any/each subcontractor.

Detailed Budget

• Personnel
• Consultant
• Equipment
• Supplies
• Travel
• Other Expenses
• Consortium/Contractual Direct Costs
• Prime Indirect Costs
• Prime Direct Costs

Budget Justification

• Narrative that fully supports and explains the basis for the information

in the Detailed Budget

• Provide sufficient detail to understand the basis for costs, the reason

that the costs are necessary, and an explanation for major cost variances

• Use the budget template to tell PCORI why the costs are reasonable

for the work to be performed

• Breakdown of costs proposed for each consortia or contractor
• Must specify any other sources of funding that are anticipated to

support the proposed research project

• Provide quotes, indirect cost rate letter, fringe benefit policy

Allowable and Unallowable Costs

NOTE

Review Appendix 3 of Application Guidelines for updates on allowability of patient and clinical care costs.

35

Budget: Commonly Denied Items

To justify budget items:

• Provide a detailed explanation for why the purchase of the

requested item(s) is required for project completion.

• Include a statement verifying the requested item(s) is not

currently available for PI use.

• Include a statement that the requested item(s) will be

purchased in accordance with applicable cost principles.

NOTE

Items purchased under PCORI-funded projects are not to be used as incentives to recruit or retain graduate students

• r other personnel.

36

Application: Common Submission Errors

• Using the wrong browser to access PCORI Online
• Use Chrome or Safari
• Not entering information into all required fields in the system
• Having multiple people working on the application at the same time
• Having the incorrect file extension
• Only PDF files can be uploaded
• Not choosing the correct document type from the drop-down menu
• Administrative Officer (AO) is unable to view the application

Tips for Success

• Adhere to the Application Guidelines for the appropriate PFA and

funding cycle

• Talk to a Program Officer if you have questions
• Start and submit application early
• Download PCORI’s Online User Manual for Submitting an Online

Application

• Ensure that all team members can see the application in the

system (check during the LOI stage)

• Inform your AO of your intent to submit
• Submit the completed application before the due date or on

it by 5:00 PM ET

38

Resources

Refer to the funding opportunities page in our Funding Center (http://www.pcori.org/funding/opportunities) for the following resources:

• PFA and Application Guidelines
• PCORI Online User Manuals
• Sample Engagement Plans
• General Applicant FAQs: http://bit.ly/applicant_faqs
• PCORI Online: https://pcori.fluxx.io/
• Research Methodology: http://www.pcori.org/node/4020

What happens to your application after you submit it?

Administrative Screening

Applicants must follow the administrative requirements stated in PCORI’s Application Guidelines. Applications may be administratively withdrawn for the following reasons:

• Exceeding page limits, budget, or time limitations
• Not using PCORI’s required templates
• Submitting incomplete sections or applications

Programmatic Screening

PCORI Mission

Applications may be programmatically withdrawn for the following reasons:

• Deviation from the approved LOI
• Inclusion of cost-effectiveness analysis (CEA)
• Inclusion of development and dissemination of clinical

practice guidelines (CPG)

• Not responsive to the program-specific PFA

42

Merit Review Process

pcori.org/content/merit-review-process

Application Review

Applications are reviewed against five criteria:

• Study identifies evidence gaps

noted in the PCORI Methodology Report or the published scientific literature

• Potential for the study to improve

PCOR methods

• Technical Merit
• Patient-centeredness
• Patient and Stakeholder

Engagement

• Applications are reviewed by a

either (1) a panel of two scientists, one patient, and one

• ther stakeholder or (2) a panel of

three scientists.

• PCORI’s Board of Governors

makes funding decisions based on merit review and staff recommendations.

Criterion 1. Study identifies evidence gaps noted in the PCORI Methodology Committee Report or the published scientific literature.

• Does the research question identify a critical gap in current

methodological understanding as noted in the Methodology Committee Report or in the scientific literature? Which particular gap(s)?

Criterion 2. Potential for the study to improve PCOR Methods

• Would the development, refinement, or comparison of methods in

this area produce more valid, trustworthy, and useful PCOR findings?

• How often would these methods be used, and how many PCOR

studies would benefit from these improved methods?

• Is the proposed approach feasible and likely to result in new

standards or in the improvement of existing standards?

Criterion 3. Technical Merit

• Is there a clear research plan with rigorous methods that demonstrates adherence to

PCORI’s Methodology Standards?

• Does the proposal delineate a clear conceptual framework/theory/model that anchors

the background literature and informs the design, key variables, and relationships being tested?

• Do the study methods reflect state-of-the-art thinking and practice in the relevant

methodological area so that results are likely to be accepted and heeded?

• Is the project timeline realistic, including specific scientific and engagement

milestones?

• Does the research team have the necessary expertise to conduct the project? Are the
• rganizational structure and the described resources appropriate to carry out the

project?

• Will the proposed methods help support the inclusion and study of diverse populations

with respect to age, gender, race, ethnicity, geography, or clinical status, or, alternatively, do the methods support the inclusion of previously understudied populations in PCOR?

Criterion 4. Patient-Centeredness

• Would the research improve processes to address questions about
• utcomes of interest to patients and their caregivers?
• Does the research address one or more of the key questions mentioned

in PCORI’s definition of patient-centered outcomes research?

• “Given my personal characteristics, conditions, and preferences,

what should I expect will happen to me?”

• “What are my options, and what are the potential benefits and

harms of those options?”

• “What can I do to improve the outcomes that are most important to

me?”

• “How can clinicians and the care delivery systems they work in help

me make the best decisions about my health and health care?”

Criterion 5. Patient and Stakeholder Engagement

• Are patients and other stakeholders engaged meaningfully in

appropriate phases of the research? Are the roles and the decision-making authority of all research partners clearly stated?

• Does the proposal demonstrate the principles of reciprocal

relationships; co-learning; partnership; and trust, transparency, and honesty?

• If engagement is deemed inappropriate in some or all aspects of

the proposed research, does the application justify why it is not?

Engagement

Effective engagement strategies can take different forms.

• PCORI’s Engagement Rubric
• http://www.pcori.org/sites/default/files/PCORI-Engagement-Rubric-

with-Table.pdf

• Sample Engagement Plans for the Methods Program
• http://www.pcori.org/sites/default/files/PCORI-Sample-Methods-

Engagement-Plans.pdf

• Engagement in Research website page
• http://www.pcori.org/content/engagement-research
• PCORI’s Methodology Standards PC-1 to PC-4
• http://www.pcori.org/assets/PCORI-Methodology-Standards1.pdf

Submission and Key Dates

What When Application Deadline November 3, 2015 by 5:00pm ET Merit Review Dates February 2016 Awards Announced April 2016 Earliest Start Date June 2016

Where can I find help?

Visit pcori.org/apply

• Application Guidelines
• FAQs
• PCORI Online User Manuals
• Sample Engagement Plans

Schedule a Call with a Program Officer

• Submit a request at pcori.org/content/research-inquiry
• Call 202-627-1884 (programmatic inquiries)
• E-mail sciencequestions@pcori.org

Contact our Helpdesk

• E-mail pfa@pcori.org
• Call 202-627-1885 (administrative and technical inquiries)

Q&A

Ask a question via the chat function in Meeting Bridge. Ask a question via phone (an operator will be on standby to take your questions).

If we are unable to address your question during this time, e-mail the Helpdesk at pfa@pcori.org.