Scoring Criteria: What (NIH) Grant Reviewers Look For in a Proposal - - PowerPoint PPT Presentation

Scoring Criteria: What (NIH) Grant Reviewers Look For in a Proposal

Jason Shepherd, Ph.D. Department of Neurobiology and Anatomy Jason.Shepherd@neuro.Utah.edu July 17th, 2019

The NIH utilizes 2 stages of review when making funding decisions

• 1. Evaluation of scientific merit in study sections
• Determine overall impact scores used to rank applications

(and percentiles)

• Articulate factors driving scores in critiques and discussion
• 2. Funding decisions by Institutes and advisory

councils based on

• Percentiles and input from study section
• Program priorities, budget, and other administrative

factors

NIH Funding Decisions

• Core Review Criteria
• Overall Impact

− Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved − Should the studies be done (Significance and Innovation) and can the studies be done (Investigator, Approach and Environment)?

Main Review Criteria

§ Significance § Investigator(s) § Innovation (NOT a specific criterion for NRSAs) § Approach § Environment

9-Point Scoring Scale

• Only ~50% of applications with top preliminary impact scores get
• discussed. Rest triaged (not discussed).
• Any member in conflict with an application leaves the room
• Three assigned reviewers state their preliminary scores
• Reviewer 1 introduces the application and presents critique (noting

strengths and weaknesses)

• Reviewers 2 and 3 highlight new issues and areas that significantly

impact scores

• All eligible members are invited to join the discussion.

At the Study Section Meeting Application Discussion

Range of Scores

• After discussion, Chair provides a summary and assigned

reviewers state final Overall Impact Scores, defining the score

• range. Final scores may differ from preliminary scores based on

discussion.

• Each panel member provides an overall impact score (online or
• n paper).
• Panel members may vote outside this range, although a reason

must be stated to the section.

5 Scored Review Criteria

• Significance

Is there a strong scientific foundation for the studies and will the studies/findings be impactful to the field?

• Investigators

Do the Investigators have the appropriate expertise and are they capable of completing the project?

• Innovation

Is there Innovation in the application?

• Approach

Is the Research Design rigorous and appropriate for the Aims?

• Environment

Will the Environment contribute to the project’s success?

Eblen MK et al. (2016) How Criterion Scores Predict the Overall Impact Score and Funding Outcomes for National Institutes of Health Peer-Reviewed Applications. PLoS ONE 11(6):

• e0155060. https://doi.org/10.1371/journal.pone.0155060

Approach is the most important criteria driving overall impact score

Overall Impact

The overall impact score is not an average of the 5 criterion scores. It reflects the likelihood that the project will significantly advance the field, if all of the Aims are achieved. Top scored applications (1-2) are potentially transformative. Does not have to be strong in all Criteria to have major impact. However, a low score in any of the individual criteria typically reduces the overall impact score of an otherwise highly significant application, because the likelihood of success is reduced.

• Should focus on how the work will move the field.
• Prospective evaluation of future impact and retrospective evaluation
• f the foundation for the studies.
• Use this section to:

(i) State the overarching and specific goals of the proposed studies (ii) Broadly describe current knowledge (big picture of what is known) (iii) What is impeding advancement of the field (e.g. technological or conceptual limitations) (iv) How you are going to overcome these limitations (e.g. technological advancement, new conceptual framework, etc.) (v) How your new approach is going to advance the field (IMPACT). Advice: Conclude the significance section with a paragraph entitled “Outcomes and Impact”.

Criterion 1: Significance

Criterion 2: Investigators

• Is the Investigator qualified to conduct the project? Do they have

the expertise and experience to conduct the proposed studies? Highlight training, publication records and roles in the project.

Use the biosketch to tell reviewers how your technical and intellectual background, and your experiences are relevant to the proposal (Personal Statement). Tell the reviewers how you are uniquely qualified for the project.

Criterion 2: Investigators

• You can also use the Biosketch and other sections (e.g.

Resources) to briefly reinforce the strength and availability of any collaborations and cores that are especially important to the proposed work.

• This

can affect scoring

• f

both ‘Investigator(s)’ and ‘Environment’.

Technical Innovation is not crucial but the methods used should be current. Use this section to:

(i) List the technical innovations of your study (If any) (ii) Note conceptual innovation(s) (ii) If no technical innovations, try to emphasize how well - established technologies will be used to address an important question or lead to conceptual innovation and discovery Do you have a unique way of viewing a longstanding problem or system? Has the problem you will address been overlooked? Might you be able to provide fresh insight that will change how we think about a system?

Criterion 3: Innovation

• Experiments should be well designed to test the specific

hypotheses and achieve the specific aims.

• What Reviewers are looking for in this section:

(i) Use of current methodological approaches is always a

• plus. Regardless of

how fancy the techniques, the methods should be current and adequate to achieve the goals. (ii) Demonstration of Feasibility. The importance of Preliminary Data. Two goals of preliminary data: a) To demonstrate feasibility but too early for significant results b) To show some preliminary results that support your hypothesis.

No matter how significant and well designed a proposal is, it will not get a high score if you do not, at a minimum, provide preliminary data to demonstrate feasibility (demonstrate that you can do it).

Criterion 4: Approach

What else are Reviewers looking for in the Approach section?

Independence of the Aims: related but not dependent on each other’s outcome. e.g. If all aims depend on the outcome of Aim 1, should Aim 1 fail the entire proposal is doomed. Potential Problems and Alternative Strategies. Identify what could go wrong and if that happens what you would do instead. Note: If you do not address this, the Reviewer will do it for you, and penalize you for not having thought of potential problems with your experimental design. Provide just enough experimental detail for reviewers to understand what you plan to do. A grant is not a paper’s Methods section. Realistic aims and timelines. Don’t be overly ambitious.

Does the PI have access to the appropriate resources in their existing environment? Does PI have access to necessary equipment, colleagues, facilities (animals, core, etc). This needs to be addressed in the Facilities and Equipment.

Criterion 5: Environment

The most important thing to know about the reviewer is that he/she has a pile

• f 8-12 grants to read (reviewer has a lot of work to do) and that only a few that

are ranked at the top of the pile (scored 1-3) have a chance of reaching the payline (competition is fierce).

Make things EASY for the review!

You loose points if:

• you make the Reviewer work too hard
• appear to be hiding issues
• your proposal is irritating:
• dense margin-to-margin text
• too small figures
• endless typos, poor grammar
• missing or mis-numbered figures, missing figure legends
• cut and pasted text from another unrelated application

without modification

• incorrect references

How to make the job easy for the NIH Reviewer

• Be concise, avoid too much jargon and excessive abbreviations.
• Make sure your grant is not too dense (leave white spaces)
• Avoid too much technical details or extensive review of the literature

Specific Aims*

(the “elevator pitch” for your grant)

• Length: 1 page
• Style: Non-technical. Write this section for all study section members,

since they’ll all read it. Note, this may be the ONLY part of your grant that reviewers (except for the 3 designated ones) will read.

• This section must include everything that is important and exciting about

your project – but without a lot of detail. * A great resource for writing NIH grant applications is The Grant Application Writer’s Workbook by Stephen Russell and David Morrison. It is available online at www.grantcentral.com.

Specific Aims (cont’d)

• The flow of logic must be so clear and compelling that reviewers at the

study section meeting will be able to follow it, even when someone else is talking to them at the same time.

• Together with the Significance and Innovation subsections, it is one of the

most important parts of the application in terms of capturing the attention of the majority of reviewers.

Specific Aims:

Introductory Paragraphs

• Develop a compelling argument for funding.

– The secret to creating a compelling flow of logic in this section is to appropriately link its components, one to another. – Begin with an interest-grabbing sentence that immediately establishes the relevance of your proposal to your field. − Describe the scope of the problem (such as a particular disease or relevance to the field). − Describe the gap in knowledge that your project will address (that is, from a research perspective, what we don’t know that we need to know in order to move forward; provides rationale for specific aims).

Specific Aims:

Introductory Paragraphs (cont’d)

– State your long-term goal. − It should be broad enough to give the impression that this study is part

• f a larger research plan that will continue beyond the bounds defined

in the Specific Aims. − It should reflect your “niche” area of research (that is, the area in which you will be the acknowledged expert). − It must be realistic (i.e., something that is clearly achievable over a finite period of time).

• For example, if you are a cancer researcher, it would not be credible

to write that your long-term goal is to cure cancer.

Specific Aims:

Introductory Paragraphs (cont’d)

– State the objective of this application − This component defines the purpose of your application, which is to fill the gap in knowledge identified in the 1st paragraph. − This must also link to your long-term goal as the next logical step along a continuum of research. − Emphasize the “product” of the research, not the “process” that produced it.

• For example, “to study” something would not be an appropriate

goal; what you want is what the study will produce.

Specific Aims:

Introductory Paragraphs (cont’d)

• If your project is hypothesis-driven, state your central hypothesis or

model you will test. – Your central hypothesis must link to the objective, because the

• bjective will be accomplished by testing your hypothesis.

− The purpose of the hypothesis is to provide focus for your research project and, therefore, your grant application. – Tell reviewers how your hypothesis was formulated (rationale) – either

• n the basis of your own preliminary data or on the published work of
• thers.

Specific Aims

• After the introductory paragraph:
• Each aim should consist of a title and a few sentences: be concise and

concrete; clarity is the goal.

• Keep the number of aims to a minimum (2 or 3).
• Aims should be able to “stand alone”: they can be related but must be

independent (i.e., they do not depend on a particular outcome of a previous aim).

• Start with rationale for the aim.
• Briefly describe how you will test the aim, details of methods is not

important here but describe essential reagents or novel techniques.

• End with what you expect to understand/know if the aim is successful: i.e.

the expected outcome.

Specific Aims:

• End with a statement that tells reviewers what will become possible after

the research is completed that is not possible now. – The gap in knowledge discussed above represents a problem because its continued existence blocks the next step in the field from being taken. – – Once the proposed research has been completed, you will be able to take the blocked step – that is why you want to do the research. – This is where you can excite reviewers: this rationale can truly be exciting because it conveys that the expected outcomes will clearly advance your field.