What Can We Learn and Apply from Journal Peer Review Rick Reiss - - PowerPoint PPT Presentation

1

What Can We Learn and Apply from Journal Peer Review

Rick Reiss Exponent Alexandria, VA

2

• What are the

advantages and disadvantages of journal and Agency peer reviews?

• Can journals teach us

anything about Agency peer reviews?

3

Issues in the Peer Review Process

• How do we select peer reviewers?
• How can we ensure a balance of perspectives?
• Does the conflict process exclude those with real

experience with chemicals?

• How should the public and stakeholders be considered

in the peer review process?

• How do we ensure that Agencies adequately respond

to peer reviews?

• How do we encourage a more robust review and

evaluation process?

4

Goals for Selection of Peer Review Panel

• Covers all of the technical areas of relevance
• Can be broad with chemical assessments
• Achieves a balanced perspective of views
• Resulting review should reflect views that a majority of

scientists would share

• Includes reviewers that are free to conclude that the

assessment is good or bad

• Viewed as credible
• Though, those unhappy with result may always complain

5

How Journals Review Papers

Review Revise Accept Reject Major Minor

Editor Review

Publish

6

Issue Agency Peer Review Journal Peer Review Peer reviewers – conflicts of interest Extensive review and disclosure Disclosure but no means of confirming information Peer reviewers – balance of perspective Large panel size allows different disciplines and can allow balance of perspective Some balance possible, but limited by 2-3 reviewers Peer reviewers – pressure to reach particular conclusion Possible pressure, given that it’s a public process No, because peer review is anonymous

7

Issue Agency Peer Review Journal Peer Review Peer reviewers – interactions Can interact with authors and other reviewers – good for information exchange and avoiding errors Cannot interact with authors and other reviewers – good for

• bjectivity and

removing pressures Leader of process Head of panel is influential, but not usually dominant Editor is free to make decision alone based on peer review

8

Issue Agency Peer Review Journal Peer Review Result of process A recommendation to Agency that it can follow or not follow Accept or reject Revisions to assessment during review Usually does not

• ccur

Formal process to request revisions and review if response is adequate Further review? At the discretion of the Agency to call panel back (or form new one) Only if author substantially revises paper and resubmits

9

Advantages and Disadvantages of Peer Review Panels

Advantages

• Large panel with different types of experts
• Interaction and exchange of information

Disadvantages

• Pressure of public process
• No ability to force changes in assessment
• No review after changes made in assessment

10

Advantages and Disadvantages of Journal Peer Review

Advantages

• Anonymous process
• Power to reject or accept
• Ability to ask for revisions and make sure they are

adequately made Disadvantages

• Small number of reviewers
• Lack of interaction with authors or other reviewers

11

Should Panels Make Recommendations on Future Review?

• Agree up front that this will be done
• Possible problem: Public agencies cannot necessarily empower

decision-making to a different entity

• Options could include:
• Acceptable, but should consider the following comments … (i.e.,

minor revisions)

• No re-review necessary
• Upgradable, but need to respond to detailed comments and re-

review

• Subset of panel reviews changes, possibly without meeting
• Rejected, scientific flaws are substantial
• Panel needs to reconvene to review changes

12

Different Possible Processes for Agencies to Address Comments

• Allow agency discretion for addressing comments
• Agency publishes a detailed response
• Allow panel to decide whether they need to see the

revised assessment.

• Full reconvening of panel
• Ask panelists to separately respond to revision
• Have agencies establish an independent ombudsman

to ascertain if agency has adequately responded to comments

13

Conclusions

• Peer reviews for agencies and in journals have

advantages and disadvantages

• One aspect of journal peer review that would improve

agency peer review are mechanisms to:

• Rate an assessment as acceptable, not acceptable, or

upgradeable

• Allow for a revision and response to revision process to

assure adequate changes are made based on peer review