Early-Stage Technology Development Research Tony Dickherber, Ph.D. - - PowerPoint PPT Presentation

Title of Presentation

Evaluation of NCI’s Strategy for Early-Stage Technology Development Research

Tony Dickherber, Ph.D. Center for Strategic Scientific Initiatives, NCI August 2017

2

Outline

1. Overview of the IMAT Program 2. Evaluation Design/Approach 3. Evaluation Findings 4. NIH-wide Technology Development 5. Conclusions and Next Steps

2

Title of Presentation

“Progress in science depends

• n new techniques, new

discoveries and new ideas; probably in that order.”

• Sydney Brenner

Title of Presentation

NCI Support for Technology Development

Low Cost Global Health Technologies (UH2/UH3, R01, R03) Common Fund Initiatives (e.g. Single Cell Technologies or Tissue Chip) Bioengineering Research Grants (R21, R01) Clinical Proteomics Technology Assessment (U01, U24) Nano (IRCN-U01 & Alliance-U24) ITCR (R21, U01, U24) SBIR/STTR (R41-44) Academic-Industrial Partnerships (R01) Genome Technology Program (R21, R01, SBIR/STTR)

Technology Development Pipeline

Concept Prototyping & Feasibility Demonstration Advanced Development towards Context of Use Scaling/Optimization within Context of Use Hardening and Validation Dissemination Typical NIH barrier for technology

Title of Presentation

Program Mission:

To support the development, maturation, and dissemination of novel and potentially transformative next-generation technologies through an approach of balanced but targeted innovation in support of clinical, laboratory, or epidemiological research on cancer.

Technology Development Pipeline

R43

• Feasibility study
• Clear commercial

potential

R44

• Development & (regulatory) validation
• Manufacturing & marketing plan
• Requires proof of feasibility and

commercialization plan

• Demonstration of transformative utility

Fast-Track

≤ $225k over 6m total cost support ≤ $1.5M over 2 years total cost support

Concept Prototyping & Feasibility Demonstration Advanced Development towards Context of Use Scaling/Optimization within Context of Use Hardening and Validation Dissemination Typical NIH barrier for technology

R21 R33

• Feasibility/Proof-of-principle study
• Highly innovative technology
• No preliminary data required
• Advanced development
• Demonstration of transformative utility
• Requires proof of feasibility

≤$400k over 3 years direct cost support ≤$900k over 3 years direct cost support

Two Tracks:

• 1. Molecular/Cellular Analysis Technologies (MCA)
• 2. Biospecimen Science Technologies (BST)

IMAT Program Structure

Title of Presentation

What is a “biospecimen science” technology?

• Sample Quality Control (e.g., RNALater)

– Focus on preserving the biological integrity of the molecular and cellular targets to be assessed – Spans the preanalytical time period from patient management variables, through sample procurement, immediate handling and preservation, and processing and sample preparation in advance of analysis

• Sample Quality Assessment (e.g., RIN)

– Focus on verifying/assessing the biological integrity of the molecular and cellular targets to be tested/measured

Title of Presentation

• Solicitation:

– Emphasis on innovative technology with transformative potential (i.e. high-risk, high-impact) – Focus exclusively on technology development (NOT biological/clinical hypothesis-driven research) – 100% investigator-initiated research grants

• Review:

– Special emphasis panels recruited based on focus of submissions, drawing heavily from former IMAT grantees – Milestone-based applications to quantitatively assess the capabilities of the technology (e.g., specificity, sensitivity, and speed) and characterize the improvement over state-of-the-art

Distinguishing Features of IMAT

Title of Presentation

8

IMAT New Award Distribution by FY

Total # of Awards Success Rate Fiscal Year

BST: Biospecimen Science Technologies MCA: Molecular/Cellular Analysis Technologies 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 5 10 15 20 25 30 35 40 45 BST R33 BST R21 MCA R33 MCA R21 Success Rate

Issued as PAR

Title of Presentation

General Breakdown of the IMAT Portfolio

Cancer Biology Technologies

• Molecular fingerprinting (-omic discovery)
• Molecular interactions
• Cancer modeling
• Imaging/spectroscopy probes
• Sample preparation
• Mechanobiololgy/microrheology
• et cetera…

Technologies for Clinical Treatment and Diagnosis

• Drug screening platforms
• Patient-derived tumor modeling
• Diagnostic imaging agents
• Cancer-targeting
• Drug delivery vehicles
• Point-of-care diagnostics
• et cetera…

Early Detection Screening

• Point-of-care detection
• Field sample collection and storage
• Liquid biopsy platforms
• et cetera…

Molecular Epidemiology Tools

• Population-scale analysis
• Low-resource setting point-of-care screening
• et cetera…

47% 4% 9% 40%

Title of Presentation

Active IMAT Portfolio

Application & Validation of Emerging Technologies for Cancer Research (R33)

• Optimization/scaling or other further development
• Analytical/technical validation in biological context of use

Innovative Technologies for Cancer Research (R21)

• Initial proof-of-concept
• Quantifiable milestone driven development plan

8 4 6 3 3 6 2 8 3 6 4 5 4 5 1 4 2 1

clinical diagnostics drug screening epigenomics genomics glycomics imaging immunotherapy liquid biopsy metabolomics modeling novel biosensor pathway tools proteomics sample prep sample QA single cell transcriptomics treatment

Current R21 Portfolio (75 Active Projects)

6 2 4 4 3 1 3 1 4 6 2 2 3 1 5 2

Current R33 Portfolio (49 Active Projects)

11

Outline

1. Overview of the IMAT Program 2. Evaluation Design/Approach 3. Evaluation Findings 4. NIH-wide Technology Development 5. Conclusions and Next Steps

Title of Presentation

IMAT FOA & Evaluation History

RFAs Renewed for 5 years

• 3 R21 (1 is a 3-yr award)
• 2 R33
• 2 STTR
• 2 SBIR

RFAs Renewed for 1 year

• 2 R21 (3 yr awards)
• 2 R33

RFAs Renewed for 3 years

• 2 R21 (3 yr awards)
• 2 R33

Evaluation Feasibility Study Full Program Evaluation IMAT RFAs Approved for 3 years

• 3 R21/R33
• 2 STTR/SBIR

RFAs Renewed for 2 years

• 2 R21 (both 3-yr awards)
• 2 R33

Evaluative Update Targeted Evaluation IMAT PAR Renewed

• 2 R21/R33
• 2 STTR/SBIR

IMAT PAR Renewed

• 2 R21/R33
• 1 STTR/SBIR

IMAT PAR Released

• 1 R21/R33
• 1 STTR/SBIR

Full Program Evaluation (until FY2016)

Ongoing Evaluation

RFA Renewed for 3 years

• 2 R21
• 2 R33
• Competitive Revisions

Title of Presentation

Evaluation Criteria

• number of publications that cite a specific IMAT award number;
• number of patent applications submitted to the US Patent & Trademark Office (USPTO);
• number of patent applications granted or approved by the USPTO based on patent

applications that cite a specific IMAT award number in one of four government interest fields;

• number of IMAT‐funded technologies now used in other NCI and NIH strategic

initiatives; and

• follow‐up case studies on previously funded technology development projects and

platforms, including their current use by and utility to the extramural scientific and clinical communities.

13

Title of Presentation

• Group: Ripple Effect Communications
• Design: From Macro International during a prior Evaluation

Feasibility Study for the IMAT program (2007)

– Key: Follow each technology from before target award to after to understand how the program/intervention affected the outcome

• Focus: Outcomes for all IMAT project prior to 2014

– 705 unique awards (including SBIR and STTR) – archival data records, web-surveys, and phone interviews – Included web-survey of and archival data analysis for a comparison group and phone interviews with IMAT technology end-users

14

2015-16 IMAT Evaluation Overview

Title of Presentation

Logic Model (Conceptual Framework)

15 Courtesy Ripple Effect Communications

Title of Presentation

Evaluation Objectives

16 Courtesy Ripple Effect Communications

Logic Model Area Evaluation Question Archival Data Web-based Survey Interview Initial Investment What were the pre-existing technologies that served as the basis for technology developed by IMAT? ✓ ✓ What technologies were proposed and what technologies were funded? ✓ ✓ Program Activities How did the application process, FOA/solicitation, and IMAT funding structure (mechanisms) impact the development of the technology? ✓ How were the technologies developed during the funding period for IMAT grantees? ✓ How did interactions with NIH, NCI, or other organizations impact the development of the technology for IMAT grantees? ✓ ✓ How did the research environment (e.g. institutional support; other related research activities) impact the development of the technology? ✓ ✓ Short-, Medium-, and Long-term Outcomes What was the technology development path after IMAT funding? ✓ ✓ ✓ How were the details of the technology spread to scientific and/or clinical audiences? ✓ ✓ To what extent and in what setting(s) is the technology or methodology being used? ✓ ✓ Are there common themes for those grantees that did not achieve their aims within the IMAT funding period? If so, what are the themes? ✓ Did the short-term and medium-to-long-term outcomes differ from the comparison group? Stage of Development ✓ Dissemination of technology via publications and patent ✓ Self-reported long-term impact ✓

Title of Presentation

Development of Comparison Group

17

C = specific aims contains “cancer” or synonym IN = specific aims contains “innovate” or derivatives C&IN = specific aims contains both “cancer” and “innovate”

Courtesy Ripple Effect Communications

Trans-NIH Eval Advisory Group

Title of Presentation

Development of Comparison Group

18

$0 $100,000 $200,000 $300,000 $400,000 $500,000 $600,000

R21 R33 R41 R42 R43 R44

Average Award Amount

Comparison IMAT

113 146 161 152 182 178 127 111 173 176 190 188 20 40 60 80 100 120 140 160 180 200

R21 R33 R41 R42 R43 R44

Average Priority Score

Comparison IMAT

Courtesy Ripple Effect Communications

Title of Presentation

Data Collection Overview

19 Courtesy Ripple Effect Communications

Title of Presentation

20

Evaluation Findings: Productivity

24 48 91 281 0% 20% 40% 60% 80% 100% Grants

Percentage of IMAT Grants

Total IMAT Grants (N=444 or ~73% of all awards)

Accounts for 75-100% of publications Accounts for 50-75% of publications Accounts for 25-50% of publications Accounts for top 25% of publications

1003 2097 3031 3814 4480 5646 6583 8420 9758 1114211655 2000 4000 6000 8000 10000 12000 14000 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Number of Citations

Year of Citation

6447 11939 5540 10674 5467 4865 6850 6086 4408 7246 2115 2572 678 91 24

50 100 150 200 250 300 350

Number of Publications

Fiscal Year of Grant

100 176 108 171 142146157 200 162 275 173171153 49 59 50 100 150 200 250 300 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Number of Publications

Fiscal Year of Grant

10 20 30 40 50

Number of Awards

Courtesy Ripple Effect Communications

Title of Presentation

21

IMAT vs. Comparison Group

57 53 60 70 23 6 16 12 22 69 1 11 15 65 111 2 5 100 200 300 FDA approval Clinical trials International approval Licenses Patents

Planned Submitted/Initiated Approved/Completed Rejected

45 45 43 49 31 3 15 2 11 42 6 16 10 27 51 1 1 2 3 50 100 150 FDA approval Clinical trials International approval Licenses Patents

Planned Submitted/Initiated Approved/Completed Rejected

IMAT Grant Outcomes Comparison Group Grant Outcomes

10 20 30 40 50 60 70 80 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Number of Grants Awarded Fiscal Year of Grant

IMAT Comparison

Courtesy Ripple Effect Communications

100 176 108 171 142 146 157 200 162 275 173 171 153 49 59 1 25 28 60 101 36 104 79 79 78 25 38 58 23 50 100 150 200 250 300 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Number of Publications Fiscal Year of Associated Grant

Title of Presentation

22

Self-Reported Outcomes

45 54 57 54 26 49 50 22 17 33 75 59 29 44 110 65 41 33 57 93 61 91 154 122 34 100 200 300 400

Advancement of ability to diagnose Advancement of ability to treat Improve quality of biospecimens used in clinical management Improve utility of biospecimens used in research Improve standards/methods for conducting cancer research

Number of Grants

Area of Impact

No Impact Little Impact Moderate Impact Great Impact N/A (Not a goal of this technology) 44 29 58 51 38 22 37 12 11 16 34 52 4 17 20 29 41 8 19 22 77 47 121 106 107 50 100 150 200 250

Advancement of ability to diagnose Advancement of ability to treat Improve quality of biospecimens used in clinical management Improve utility of biospecimens used in research Improve standards/methods for conducting cancer research

Number of Grants

Area of Impact

No Impact Little Impact Moderate Impact Great Impact N/A (Not a goal of this technology)

IMAT Grant Outcomes (310 responses) Comparison Group Grant Outcomes (211 responses)

Survey Responses

Courtesy Ripple Effect Communications

Title of Presentation

• Certified the value of archival data for addressing evaluation

questions

• Certified the value of survey and interview data for

developing more comprehensive appreciation for the impact

• f the program/intervention
• Recommendations:

– Make better use of the annual PI meeting – Improve options for supporting commercialization efforts – Develop standardized review requests at close of awards that includes downstream development information – Incorporate measures of risk during initial award to better quantify differences within IMAT and among comparison groups – Continue pursuing end-user interviews for evaluation of impact

23

Ripple Effect Report

24

Outline

1. Overview of the IMAT Program 2. Evaluation Design/Approach 3. Evaluation Findings 4. NIH-wide Technology Development 5. Conclusions and Next Steps

Title of Presentation

• Performance Measures study from the Science and Technology Policy

Institute (STPI) developed during 2015 [link]

• The study had three components:

– Developing a comprehensive catalog of NIH Funding Opportunity Announcements (FOAs) that are focused on technology development for achieving a specific goal – Developing case studies of a representative sample of FOAs based on discussions with FOA program officers to identify measures being used and lessons learned – Identifying candidate outcome measures for assessing technology development initiatives and a data collection infrastructure that would be required to implement these measures in a consistent and ongoing manner

25

NIH-wide Technology Development Evaluation

Title of Presentation

Tech Dev FOA Catalog

26

Type of Solicitation

FOA Type Number Percentage RFA 190 67% PA 58 20% PAR 30 11% PAS 6 2%

Activity Codes

Activity Code Number Percentage

R43/R44 76 27% R21 64 23% R41/R42 41 14% R01 37 13% R33 26 9% U01 16 6% R21/R33 12 4% R43 only 7 2% R41 only 6 2%

Also: P01, U24, U54, DP3, R03, UH2, P41

• Included 284 distinct FOAs,
• rganized into 83 distinct groups
• FOAs supported 1,956 distinct

awards

• $1.83 billion in total NIH spending
• ver 10 years

Courtesy of the Science and Technology Policy Institute

Title of Presentation

Largest and Smallest FOA Groups (by Total Cost)

Largest:

• Instrument Development for Biomedical

Applications (165 awards, $174 million)

• Revolutionary Genome Sequencing Technologies:

$1000 Genome (100 awards, $171 million)

• Clinical Proteomic Technologies for Cancer

Initiative (CPTC): Proteome Characterization Centers (61 awards, $161 million)

• Validation and Advanced Development of

Emerging Molecular Analysis Technologies for Cancer Research (187 awards, $118 million)

• Early-Stage Innovative Molecular Analysis

Technology Development for Cancer Research (233 awards, $116 million)

• National Technology Centers for Networks and

Pathways Program (6 awards, $82 million)

• Exceptionally Innovative Tools and Technologies

for Single Cell Analysis (42 awards, $66 million)

Smallest:

• New Technology to Screen for Mild Hearing Loss

in Children (1 application, no awards)

• In vivo Methods for Assessing Placental

Development and Function (2 applications, no awards)

• Development of Diagnostic Screening Test for

Salt Sensitivity (3 applications, no awards)

• Technologies To Assess Sleep Health Status in

Populations (7 applications, 1 award, $0.1 million)

• Innovative Technologies and Assays in Support of

HIV Cure Research (9 applications, 1 award, $0.3 million)

• Methods Development for Obtaining

Comprehensive Genomic Information from Human Specimens that Are Easy to Collect and Store (7 applications, 2 awards, $0.4 million)

• Development of a Vestibular Neural Prosthesis (6

applications, 2 awards, $2.7 million)

Courtesy of the Science and Technology Policy Institute

Title of Presentation

Tech Dev FOA Catalog

28

Technology Area

Stage of Development

Stage of Development Number of FOA Groups Early Only 15 Early/Intermediate 26 Intermediate Only 4 Intermediate/Late 13 Late Only 3 Early to Late 22

• Majority address a defined area within biomedical research
• r clinical care
• Majority encompass a diverse set of products
• Equally divided between technologies intended for research

versus clinical use

• Few have defined performance requirements

Categorization Number of FOA Groups Medical Devices 19 Molecular Analysis 11 Information Technology 9 Cells/Tissues Analysis 6 Point of Care Devices 6 Proteomic Analysis 6 Genomic Analysis 5 Imaging 5 Low Cost Medical Devices 5 Implantable Devices 4 Biospecimen Technologies 3 Others 4

Courtesy of the Science and Technology Policy Institute

Title of Presentation

Lessons Learned: Program Management Best Practices

• Award flexibility beneficial. Because technology development projects often require more

time and money than typical of discovery research projects, FOAs need to take advantage of

• pportunities for longer award periods and larger award sizes; multiple acceptance dates are

also valuable.

• Tailored review necessary. Because technology development projects often involve

engineering and physical sciences disciplines and have more applied goals, tailored review is essential.

• Milestones valuable. Because technology development projects are intended to result in a

defined entity for use in research or the clinic, milestones help in charting progress.

• Grantee meetings valuable. Grantee meetings open to industry, other researchers, and

potential investors are valuable for sharing information among awardees, facilitating collaborations, and exploring potential commercial relationships.

• Program officer expertise critical. Technology development program officers require three

critical characteristics: – Clear understanding of requirements for commercializing or otherwise disseminating technologies – Expertise in the technology field – Familiarity with the relevant investigator community

Courtesy of the Science and Technology Policy Institute

Title of Presentation

• Commercialization a hurdle, especially for clinical technologies.

Technologies for clinical use almost uniformly require more funding than available through standard NIH award mechanisms. As a result, clinical technologies often languish even if early-stage clinical testing has been completed.

• Funding “blue-sky” technology development difficult. Only the R21

mechanism was viewed as being tailored to fund truly high-risk projects and additional approaches for encouraging such projects need to be developed.

• Greater coordination of technology development efforts is needed.

Program officers were generally aware of other ongoing technology development initiatives, but indicated that a forum where they could share lessons learned and best practices would be beneficial.

30 Courtesy of the Science and Technology Policy Institute

Lessons Learned: Ongoing Challenges

Title of Presentation

Proposed Outcome Measures

31

• Ultimate Objectives

– Dissemination and use of the technology – Increase in the overall level of NIH-funded research activity in the technology development domain

• Intermediate Outcomes

– Achievement of technical milestones – Conversion of exploratory awards – Technology licensing or other pre-commercialization activity – Progress toward clinical use – Data and software downloads

Courtesy of the Science and Technology Policy Institute

Title of Presentation

Required Data Collection Infrastructure for Outcome Measures

• Post-award reporting

– Publications and grants reporting use of the technology – Appearance in clinical practice guidelines – Conversion of exploratory awards – Software downloads (for information technologies), and – Additional data on commercialization activities and progress toward clinical use

• Bibliometric analysis

– Number and rate of citations – Manual analysis of citations for relevance

• Publication/grant application analysis

– MEDLINE and NIH grant database searches for the name of the technology or the technology development domain or for references to publication(s)

Courtesy of the Science and Technology Policy Institute

Title of Presentation

• Commercial activity in technology development domain

– Manual searching of publicly available information based on product names – Queries to relevant NIH program officers

• Expert panels

– Assessment of whether a technology has achieved widespread dissemination and use – Assessment of whether there is increased research activity in the relevant technology development domain

• Milestone analysis

– Standard template for recording milestones in grant applications – Standard Research Performance Progress Report (RPPR) template for reporting progress toward milestones

Required Data Collection Infrastructure for Outcome Measures

Courtesy of the Science and Technology Policy Institute

34

Outline

1. Overview of the IMAT Program 2. Evaluation Design/Approach 3. Evaluation Findings 4. NIH-wide Technology Development 5. Conclusions and Next Steps

Title of Presentation

Program Mission:

To support the development, maturation, and dissemination of novel and potentially transformative next-generation technologies through an approach of balanced but targeted innovation in support of clinical, laboratory, or epidemiological research on cancer.

Technology Development Pipeline

R43

• Feasibility study
• Clear commercial

potential

R44

• Development & (regulatory) validation
• Manufacturing & marketing plan
• Requires proof of feasibility and

commercialization plan

• Demonstration of transformative utility

Fast-Track

≤ $225k over 6m total cost support ≤ $1.5M over 2 years total cost support

Concept Prototyping & Feasibility Demonstration Advanced Development towards Context of Use Scaling/Optimization within Context of Use Hardening and Validation Dissemination

R21 R33

• Feasibility/Proof-of-principle study
• Highly innovative technology
• No preliminary data required
• Advanced development
• Demonstration of transformative utility
• Requires proof of feasibility

≤$400k over 3 years direct cost support ≤$900k over 3 years direct cost support

Competitive Revisions

• Validation within the context of a

compelling hypothesis

• Pursued in collaboration with

end-users ≤$300k over 2 years direct cost support (R01, U01, U54, P01, P50)

New IMAT Program Structure

Title of Presentation

• NCI has approved IMAT to continue making new awards through

2020 & Cancer MoonshotSM Initiative has a focus on New Enabling Technologies

• NIBIB reissued EBRG/BRG/BRP in 2016
• NHGRI novel NGS RFAs and genome technology PARs still open
• NIGMS launched an exploratory R21/R01 tech dev program in 2017
• NINDS & NIMH coordinated BRAIN Initiative efforts issued a variety
• f research and training efforts in 2017
• NIAID is currently deliberating launch of their own version of IMAT
• …time to initiate a P&E version of the NIH-wide Bioengineering

Consortium (BECON)?

Next steps…

37

Questions? (301) 547-9980 anthony.dickherber@nih.gov