Welcome Session 2 PCORI Annual Meeting November 2, 2018 - - PowerPoint PPT Presentation

Peer Review Workshop: Three Part Series The Patient Voice in Peer Review

Session 2

PCORI Annual Meeting November 2, 2018

Welcome

#PCORI2018

2 • November 2, 2018

Purpose of this Workshop

• Demonstrate the importance of patient reviewers’ input and the

impact on the PCORI Final Research Report

• Explain why patient peer review is a worthy endeavor
• Address what it takes to be a patient reviewer
• Answer your questions
• Provide information about:
• How to get involved in peer review
• Resources for patient reviewers

3 • November 2, 2018

Patient Peer Reviewers include:

• Patients
• Patient Advocates
• Unpaid Caregivers
• Family Members

Introduction: This is a Big Deal

PCORI has a unique peer review process that involves patient reviewers

Amy Price, PhD Patient Editor for Research and Evaluation The British Medical Journal Marina Broitman, PhD Associate Director, Peer Review PCORI Whitney Brower, MPH Program Associate, PCORI Public and Patient Engagement PCORI

#PCORI2018

5 • November 2, 2018

PCORI’s Obligations under its Authorizing Law

• Conduct peer review of primary research to assess:
• Scientific integrity

Do the results support the Conclusions?

• Adherence to PCORI’s Methodology Standards
• To meet these obligations, PCORI requires a Final

Research Report, which goes through peer review

6 • November 2, 2018

Getting the Word Out

• Step 1: Peer review our primary research. Patients/other stakeholders join

scientists as reviewers to help us assess how useful results will be in real life.

• Step 2: Factor results of the peer review process into planning

dissemination and implementation.

• Results’ summaries for professional and general audiences
• Spanish/audio versions
• Work with patients, family caregivers, health communications experts,
• thers to design and test content and formats
• Partner with stakeholders to tailor content and dissemination channels
• Evaluate as we go

7 • November 2, 2018

Types of PCORI Peer Reviewers

Associate Editors

Patients

(Patient Advocates, Unpaid Caregivers)

Stakeholders Subject Matter Experts Methodologists/ Statisticians

8 • November 2, 2018

Editorial Facts: Patient Peer Reviewers

183 Complete Patient Peer Reviews by 134 reviewers

• Since October 2016, PCORI

has sent out 395 invitations for review to 280 patient reviewers in our database

• 37 patients reviewers have

completed more than 1 peer review

• Some people decline to

review, some do not respond, and others have agreed but not yet completed a review

Topics patients have reviewed include:

9 • November 2, 2018

• Engagement in healthcare
• Musculoskeletal system

diseases

• Complementary and

alternative medicine

• Diabetes
• Behavioral medicine
• Chronic pain
• HIV
• Cancer
• Transgender care
• Reproductive and urinary

system diseases

• Decision support tools
• Development of

educational materials

• Pregnancy and childbirth
• Mental health and

behavioral disorders

• Preventative care
• Cardiac health
• Disparities in healthcare
• Racial and ethnic minority

healthcare

• Care coordination
• Kidney disease
• Respiratory disease
• Blood disorders

10 • November 2, 2018

Partners in Patient Peer Review

The British Medical Journal was the first peer reviewed medical journal that included patients in their peer review

• process. PCORI is one of the first
• rganizations in the United States to

include patients in peer review.

Amy Price, PhD Patient Editor for Research and Evaluation, The British Medical Journal

Key Players of PCORI Peer Review: A Panel Discussion

From Peer Review, to Editor’s Response, to the Final Research Report

Patient Peer Reviewers: Barbara Sheehan, MEd Jeffrey Oliver, MBA Paul McClean Principal Investigator: Hanan Aboumatar, MD, MPH Johns Hopkins University Associate Editor: Kelly Vander Ley, PhD Oregon Health and Science University Moderator: Ilya Ivlev, MD, PhD Oregon Health and Science University

12 • November 2, 2018

The PCORI Peer Review Process

Final Research Report

(Principal Investigator)

Synthesis Letter

(Associate Editor)

Peer Review

(Patients, Stakeholders, Methodologists, Research Expert, Statistician)

13 • November 2, 2018

The Synthesis Letter

• Associate Editors provide a synthesis of reviewer comments, as

well as their own review of the report.

• The synthesis letter includes a section specifically devoted to the

patient perspective.

• In addition, Primary Investigators are asked to address all

reviewer comments in a response letter.

14 • November 2, 2018

Panel Discussion

Questions for the panelists?

15 • November 2, 2018

Panelists

• Patient Peer Reviewers:
• Barbara Sheehan, MEd
• Jeffrey Oliver, MBA
• Paul McClean
• Principal Investigator:
• Hanan Aboumatar, MD, MPH, Johns Hopkins University
• Associate Editor:
• Kelly Vander Ley, PhD, Oregon Health and Science University

Call to Action

Let’s peer review!

Rebekah Webb, MPA Project Coordinator, Editorial Office for PCORI Peer Review Oregon Health and Science University

#PCORI2018

17 • November 2, 2018

Impact of Patient Review on Final Report

Example A: Prescription Pain Management (Opioid Therapy)

• The patient reviewer commented that the study is framed in the

context of overall opioid dose reduction, and suggested that alternative pain therapies could replace opioid treatment instead of presenting them as adjunct to pain control.

• Also expressed was concern that the development of the intervention

did not include stakeholders that do not share beliefs about the inherent risks of long-term opioid treatment.

18 • November 2, 2018

Impact of Patient Review on Final Report

Example A: Prescription Pain Management (Opioid Therapy)

→ The authors improved the report with the following

additions:

• A more balanced background on the pros and cons of

long-term opioid treatment

• Clearer description of curriculum intent
• Greater transparency of stakeholder positions and the

underrepresentation of liberal perspectives on long-term

• pioid use

19 • November 2, 2018

Impact of Patient Review on the Final Report

Example B: A Multidisciplinary Approach to Oncology

• In the Discussion & Conclusion section, the patient

reviewer suggested describing, from a patient’s point of view, what the difference is between the intervention (a multidisciplinary clinic) and a standard care clinic.

20 • November 2, 2018

Impact of Patient Review on the Final Report

Example B: A Multidisciplinary Approach to Oncology

→ The authors added a description of the flow of patient appointments through a multidisciplinary clinic and also provided a description of the standard care appointment flow in their healthcare system.

21 • November 2, 2018

Impact of Patient Review on Final Report

Example C: Clinical Effectiveness of Clinical-Community Childhood Obesity Interventions

• The Reviewer thought the report needed more

discussion of patient/stakeholder engagement--how they were identified and what they identified as primary concerns.

22 • November 2, 2018

Impact of Patient Review on Final Report

Example C: Clinical Effectiveness of Clinical-Community Childhood Obesity Interventions

→ The authors added a two-page description of how stakeholders were identified, how the advisory boards were comprised, and what specific outcomes the advisory boards determined were most important.

23 • November 2, 2018

Impact of Patient Review on Final Report

Example D: Family Navigator Services for Children Treated with Antipsychotic Medication

• The reviewer asked what research aims were created

based on information gathered through interviews and focus groups.

24 • November 2, 2018

Impact of Patient Review on Final Report

Example D: Family Navigator Services for Children Treated with Antipsychotic Medication

→ In response, the authors added how they sought consumer and stakeholder input over a 6-month, in-depth process to create their research questions.

25 • November 2, 2018

Impact of Patient Review on Final Report

Continued- Example D: Family Navigator Services for Children Treated with Antipsychotic Medication

• The reviewer thought it was important to include

examples of how Family Navigators could affect overall health of patients.

26 • November 2, 2018

Impact of Patient Review on Final Report

Continued- Example D: Family Navigator Services for Children Treated with Antipsychotic Medication

→ In response, the authors included more case examples relating to the Family Navigator in their final report.

27 • November 2, 2018

Seeing like a patient

In these examples, Research Teams did a tremendous amount of work with patients/caregivers during their research…. . . . But that was not communicated well in their research report until the patient reviewer asked for more details during the peer review process. Having patients involved in research during all phases of the research process makes a difference!

28 • November 2, 2018

Seeing like a patient

Patient reviewers

Help the reports be more patient- centered while still being scientifically rigorous Help identify jargon or assumptions that need clarification from the author(s)

29 • November 2, 2018

Exercise: More Helpful and Less Helpful Patient Peer Reviewer Comments

In this exercise, we will examine a set of Patient Peer Reviewer comments that were taken from actual reviews.

• 1. Please read the Review Form Question and the Patient

Reviewer Response on the Worksheet

• 2. Decide if you think it is a More Helpful or Less Help
• 3. You can share your reasoning with the group (optional)
• 4. Suggest how this comment could be improved, if applicable

30 • November 2, 2018

More Helpful or Less Helpful Comment?

Question (Reviewer Form- Background): Are the study aims and/or research questions meaningful to you or patients with similar health experiences? Patient Reviewer Comment A The report has a lot of undefined terminology that I did not understand.

31 • November 2, 2018

Less Helpful Comment (A)

Reasoning: This is a less helpful comment because the reviewer has not specified which terminology they did not understand. It is helpful to bring up content that needs further definition or explanation, but unless the author knows specifically what the reviewer is concerned about, it is difficult to determine how to improve the research paper.

32 • November 2, 2018

Less Helpful Comment (A)

→ Editor’s Advice:

• The best comments a reviewer can provide are constructive

with specific examples of where improvement is needed.

• This comment would be strengthened if the reviewer had

identified what terms were confusing and where the terms were used in the paper.

• Your opinions and generalizations will need evidence to

support them.

33 • November 2, 2018

More Helpful or Less Helpful Comment?

Question (Reviewer Form “Patient/Stakeholder Engagement and Partnership”): Does the report adequately describe how patients, caregivers, or members of patient advocacy organizations were consulted and how their input influenced the planning and conduct of the study? Patient Reviewer Comment B Please explain why there is an uneven distribution of survivors and clinicians in part 1.

34 • November 2, 2018

More Helpful Comment (B)

Reasoning: This is a very specific request about an outcome that was unclear to the reviewer. The principal investigator (PI) either needs to address this concern and explain their reasoning or list it as a limitation. The PI could say more about how patients and other stakeholders were involved, including both the strengths and limitations of the approach used.

35 • November 2, 2018

More Helpful Comment (B)

→ Editor’s Advice: These types of comments help the PIs to communicate their results on a level that resonates with a larger audience.

36 • November 2, 2018

More Helpful or Less Helpful Comment?

Question (Reviewer Form “Discussion & Conclusion”): Does the report provide information that will help patients, caregivers, or clinicians make decisions? Patient Reviewer Comment C The second point on page XX reports that "Most women reported having complete trust in providers" - and yet also reports that they felt "intimidated". This was confusing to me because it sounds as though the women were intimidated into following their providers' recommendations, rather than having "complete trust" in their survivorship surveillance plan. It would be helpful to "hear" more comments from the women in the focus groups.

37 • November 2, 2018

More Helpful Comment (C)

Reasoning: The comment shows the authors where they need to be clearer in presenting the data. It also indicates that the subset of comments that the authors provided from the focus group is not sufficient for readers to understand what was done in the study.

38 • November 2, 2018

More Helpful Comment (C)

→ Editor’s Advice:

• Comments that point out specific confusions or things needing

further explanation are helpful to the authors.

• Asking for more examples from the data is also appropriate, as

it is often hard for the author to know how much information to provide. In answering this question, consider what specific information a patient or stakeholder might need when making a decision about this topic in the real world. If that information is not included, say what is missing.

39 • November 2, 2018

More Helpful or Less Helpful Comment?

Question (Reviewer Form “Overall”): Please share your overall assessment of the report or any other comments on this research and this report. Particularly consider how the report or the research might be improved. Patient Reviewer Comment D I see no ways in which the report or underlying research might be improved.

40 • November 2, 2018

Less Helpful Comments (D)

Reasoning: This comment is less helpful because it gives no justification for the reviewer’s opinion of the research. It doesn’t identify if the research cannot be improved because it is perfect as-is, or because it is so bad that there is no hope of improving it.

41 • November 2, 2018

Less Helpful Comments (D)

→ Editor’s Advice:

• The best comments a reviewer can give are supported

with evidence.

• Simply agreeing or disagreeing with the questions

posed on the peer review form is less helpful than stating your opinion and providing evidence of examples from the report to support your claims.

• Be as specific as possible.

42 • November 2, 2018

Join our pool of patient peer reviewers

www.sciencesupport.org/ PCORIpeer

All reviewers will receive a $50 honorarium for a completed review.

43 • November 2, 2018

Upcoming 2019 reports focus on:

• Addiction/Substance

Abuse

• Mental/Behavioral

Health

• Cardiovascular

Diseases

• Respiratory Diseases
• Functional Limitations

and Disabilities

• Infectious Diseases
• Low-income and

underserved populations

• Racial and ethnic

minorities

44 • November 2, 2018

Thank you!

Ilya Ivlev, MD, PhD Kelly Vander Ley, PhD Marina Broitman, PhD Rebekah Webb, MPA Whitney Brower, MPH Special thanks to Amy Price, PhD Barbara Sheehan, MEd Hanan Aboumatar, MD, MPH Jeffrey Oliver, MBA Paul McClean

questions?

45 • November 2, 2018

Learn More

• www.pcori.org
• peerreview@pcori.org
• #PCORI2018
• Editorial Team: pcoripeerreview@sciencesupport.org
• Apply to Review: www.sciencesupport.org/PCORIpeer