Improving Implementation with the SPeeding Research INTerventions - - PowerPoint PPT Presentation
Improving Implementation with the SPeeding Research INTerventions (SPRINT) Program Anna Gaysynsky, MPH Alisa Ainbinder, PhD, MA, April Oh, PHD, MPH, Tara Loomis, MBA, MA Cynthia Vinson, PHD, MPA 10th Annual Conference on the Science of
Anna Gaysynsky, MPH Alisa Ainbinder, PhD, MA, April Oh, PHD, MPH, Tara Loomis, MBA, MA Cynthia Vinson, PHD, MPA
10th Annual Conference on the Science
Arlington, VA December 5, 2017
2
Spring of 2017)
both quantitative and qualitative data from SPRINT participants (n=20 teams consisting of 2-4 team members).
available)
and the relationship between D&I and commercialization
3
4
*Note: response options differed slightly in the two versions of the survey
45% 45% 9% 0% 0%
0% 10% 20% 30% 40% 50% 60%
Overall, how would you rate the SPRINT course?
Excellent Very good Fairly good Not very good Not good at all 44% 48% 8% 0% 0%
0% 10% 20% 30% 40% 50% 60%
Overall, how would you rate the SPRINT course?
Very good Good Fair Poor Very poor
5
with facilitators as “Good/ Excellent”, and felt their feedback was helpful (86%)
with facilitators as “Good/Very good”, and felt their feedback was helpful (75%)
*Note: response options differed slightly in the two versions of the survey
“Although it was uncomfortable at times, the instructors' insistence that we challenge our assumptions was important.”
50% 42% 33% 33% 17% 21% 4% 0% 10% 20% 30% 40% 50% 60% Quality of interactions with facilitators Feedback from facilitators on ideas
Instructor Ratings
Very good Good Fair Poor Very poor 36% 36% 50% 50% 9% 14% 5% 0% 10% 20% 30% 40% 50% 60% Quality of interactions with faculty members Feedback from faculty on ideas
Instructor Ratings
Excellent Good Fair Poor Very Poor
6
would recommend the course to other researchers
would recommend the course to other researchers.
86% 14% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Would you recommend SPRINT to other eligible researchers?
Yes Unsure No
84% 12% 4% 0% 20% 40% 60% 80% 100%
Would you recommend SPRINT to other eligible researchers?
Yes Unsure No
7
that gets a R01 should be required to go through this process before they begin their research work.
start a business/start up. I do not think it is well suited for researchers who want to implement their interventions into real world clinical settings. Perhaps a program like TIDIRH would be better for the latter.
devote 20% time for the 6-7 weeks. Which is fine but needs to be built into clinical, etc duties…
because of the pace. Each team member spent over 8-12 hour/week on SPRINT- related activities
8
10)
interviews [Data not available for Cohort I]
course and what (if anything) should be done about it.
schedule the time (personally/professionally)
9
While many participants felt that the time commitment was challenging, in the end, most participants felt SPRINT was a valuable and worthwhile learning opportunity.
I thought the SPRINT program was terrific! It was a lot of work in a short amount of time but I'm very thankful that I was chosen to participate as I learned a great deal about the current product/intervention and skills/experiences that I can incorporate into future interventions. … the course was intense and the materials were challenging. I worked a lot harder (put in more time) than I had expected. But there is no easy way to teach difficult materials; there is no easy way to push people outside of their comfort zone…I learned a great deal and I'm very grateful for the opportunity.
10
71% 25% 4% 0% 0% 0% 0% 50% 100%
Obtained new skills/knowledge as a result of participation in SPRINT.
Totally agree Agree Neither agree nor disagree Disagree Totally disagree Doesn't apply/No response 50% 42% 8% 0% 0% 0% 0% 20% 40% 60%
Feel confident applying the skills/knowledge developed during the course
Totally agree Agree Neither agree nor disagree Disagree Totally disagree Doesn't apply 67% 33% 0% 0% 0% 0% 50% 100%
Obtained new skills/knowledge as a result of participation in the program
Strongly agree Agree Neutral Disagree Strongly disagree 52% 43% 5% 0% 0% 0% 20% 40% 60%
Feel confident applying the skills/knowledge developed during the institute
Strongly agree Agree Neutral Disagree Strongly disagree
11
general... I never saw it [before] as selling a product… I look forward to applying some of things I learned in my personal research.
for the future- so definitely it’s a skill development
exciting.
implementing interventions in ways that, given an academic position, you tend not to think about”…it’s been very helpful…to learn some of those real world barriers and challenges that exist in implementing an intervention
pitfalls- because you can think through the whole process in advance…
12
and people who decide about business and investment... this was a good crash course in how to engage those stakeholders and figure out what questions we should be asking... Because what information they need to make their various decisions – [is] not in
us and had the same perspectives. But by the end of the course we were interviewing people totally outside of our fields. That’s been transformative.
world… I was completely clueless about that and intimidated by it…[but] I enjoyed finding common ground with people that we’re going to have to have relationships with if we’re going to impact public health through the dissemination of our intervention.
making sure it actually meets the needs and demands of practitioners and patients
meeting people that do the same work I do- but not researchers - different stakeholders that all have the same common goal. If it wasn’t for this program- I wouldn’t be invited to sit at some of the tables I’ve now been invited to sit at in the last 6 weeks…
13
significantly more that the required minimum of 30.
interview requirement
interviews where our time could have been better used-but it was just that desperateness to get something in.
wanted to take a step back and think about, “what questions do I need to ask and get answered?” “Where do I really need to go?” “Where do I want to focus?” Whereas the push was really for numbers and just, "go do it"
14
Note: Cohort II survey did not ask about receiving royalties or contributing ideas utilized by a company
2.1 75.4 0.3 0.2 0.5 56.6 5.0 2.6 0.3 2.7 1.2 0.5 51.0 0.2 0.0 14.7 0.0 0.2 3.3 5.3 0.3 3.1 0.9 4.8 0.4 0.3 10 20 30 40 50 60 70 80
Average number of times participants reported participating in various experiences prior to SPRINT
Cohort I Cohort II
15
Revenue streams (59%), & Cost structures (58%). Participants were more familiar with Customer segments, Key Partners, & Customer relationships, with fewer than half indicating they knew “nothing/very little” about them.
model component, but while most felt they had a “moderate amount” or “great deal” of knowledge about most of the components, a substantial proportion of participants still felt they only had “some” knowledge about Cost Structure (29%) and Revenue Streams (23%), indicating a need to spend more time on those components.
0% 20% 40% 60% 80% 100%
Channels Cost Structure Customer relationships Customer segments Key activities Key partners Key resources Revenue streams Value propositions Channels Cost Structure Customer relationships Customer segments Key activities Key partners Key resources Revenue streams Value propositions Before After
Cohort I Knowledge Change
Nothing Very little Some A moderate amount A great deal
16
participants, along with Key partners and Key activities. More than 85% of respondents knew “Very little/ Nothing” about Revenue streams, Cost structures, & Value propositions.
but some uncertainty remained in regard to Key resources, Cost Structure, & Revenue Streams
0% 20% 40% 60% 80% 100%
Customer Segments Channels Value Propositions Key Partners Customer Relationships Key Activities Key Resources Cost Structure Revenue Streams Customer Segments Channels Value Propositions Key Partners Customer Relationships Key Activities Key Resources Cost Structure Revenue Streams Before After Cohort 2 Knowledge Change
Nothing Very little Not sure Some A great deal
17
90% 67% 67% 100% 38% 90% 83% 42% 25% 63% 13% 83% 10% 33% 33% 0% 62% 10% 17% 59% 75% 38% 88% 17% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Assessing readiness for implementation Identifying a viable commercialization path Developing a scalable business model Identifying partners Getting investors interested Identifying & validating key customer segments Assessing readiness for implementation Identifying a viable commercialization path Developing a scalable business model Identifying partners Getting investors interested Identifying & validating key customer segments Cohort I Cohort II
Percent of participants who felt they learned "a moderate amount/a great deal" vs. "nothing, some, or a little" about different aspects of commercialization
A great deal/A moderate amount Nothing, A little, or some
18
33% 57% 52% 33% 57% 57% 38% 43% 57% 19% 10% 5% 5% 10% 14% 0% 0% 0% 0% 10% I was able to identify a market for my behavioral intervention I gained an appreciation for what it takes to commercialize an intervention & the barriers to intervention adoption I gained a toolset I can use to make my research more “stakeholder” focused and commercially viable I increased my understanding of the real value of my intervention from a market perspective I was able to expand my network as a result of the interactions I had with teachers, investors, customers, and/or mentors 0% 10% 20% 30% 40% 50% 60%
Extent of participant agreement with obtaining promised benefits (Cohort I)
Strongly agree Agree Neutral Disagree Strongly disagree
19
62% 38% 24% 62% 52% 38% 57% 29% 24% 29% 24% 38% 33% 29% 5% 19% 24% 5% 14% 5% 10% 10% 5% 10% 14% 10% 10% 14% 10% The program has increased my interest in further developing/refining the intervention The program has increased my interest in working with a technology based start-up I will use information and ideas from the course in my teaching I will use information and ideas from the course to help guide the development of
I will use information and ideas from the course in my career I will use the information and ideas from the course in my research program This program has enhanced my capabilities to plan and design my interventions for implementation 0% 10% 20% 30% 40% 50% 60% 70%
Impact of Sprint on Cohort I Participants
Strongly agree Agree Neutral Disagree Strongly disagree Not Applicable
20
54% 29% 79% 71% 67% 58% 42% 25% 21% 29% 25% 33% 4% 21% 4% 8% 17% 4% 8% 0% 10% 20% 30% 40% 50% 60% 70% 80% The program increased my interest in further developing/refining the intervention I will use the information and ideas from the SPRINT course in my teaching I will use information and ideas from the SPRINT course to help guide the development
I will use the information and ideas from the SPRINT course in my career. I will use the information and ideas from the SPRINT course in my research program. This program has enhanced my capabilities to plan and design my interventions for implementation.
Impact of Sprint on Cohort II Participants
Totally agree Agree Neither agree nor disagree Disagree Totally disagree Doesn't apply
21
plan to apply for an SBIR award
plan to seek nonfederal funding
product/program
plan to apply for an SBIR award
plan to seek nonfederal funding
product/program
*Note: response options differed slightly in the two versions of the survey
22
intervention is commercializable
an intervention that closely resembles the intervention they started with
(or will be) substantially different from the intervention they started with
not commercializable and 3 were unsure.
product closely resembles the intervention they started with
changes/tweaks based on the customer discovery interviews, and one noted that the team would “likely make a number of modifications to our intervention” if they weren’t already actively testing the intervention in the R01
*Note: this information was gathered as part of one item in Cohort I survey and as two separate items in the Cohort II survey, so they are not directly comparable
23
customers in an iterative process
intervention meets the payors’ needs (if different from those of the end user)
given their extensive network and incentives for meeting vaccination rates.
to develop more targeted models for different price points.
*Cohort I survey did not ask about pivots
24
meeting (e.g. a webinar covering the interview process)
commuting time and facilitate interviews.
the week's activities rather than follow them (change flipped classroom model)
participants prior to the course
the team roles
earlier so interviews that happen during the kickoff are more useful
are here for the kickoff meeting
(change flipped classroom model)
software (which can be glitchy & not intuitive)
do the weekly presentations on-on-one or create smaller groups and allow teams to give each other feedback)
25
speed things up as we were all often interviewing [the] same people
networking opportunities between SPRINT groups.
more time to talk with the other teams in a social setting
groups to work together during the initial session.
activities might be helpful
And we didn’t get to capitalize on learning from one another. It was a missed opportunity.
*Note: response options differed slightly in the two versions of the survey
26
intervention
be ideal for people with funded R21s or even for people in the grant development stage.
thinking about is going to fit out in the world. It’s almost like this [course] shouldn’t be on the back end of your training- it should be way in the front- not only in your training as a scientist- but [when you get] money from NCI- what are you going to do with that? “Oh. I’m going to write a paper.” “No, what are you going to do with the intervention?” You can’t really know until you know who you’re seeking to intervene with- and what do they actually need.
*Note: response options differed slightly in the two versions of the survey
27
28
Implemented
Under Discussion
29
30
this?” and all of us were kind of looking at him like, “What?” because we’ve invested so many years being academics… do you want us to abandon the long pursuit
entrepreneur- here was always that friction [with] thinking about developing a business- that’s not my goal- that’s not what I want... My dissemination plans are going to be a whole lot different now. And that’s exciting. It will be unique and innovative and that will be fun - but it’s not going to be about starting a business. I like my day job.
time] developing this intervention. I would like it to make it out to the real world to help people…
business…I was so science oriented. But in public health the two have to come together…This course really reaffirms those things.
31
intervention out there to someone else? Something we’ve spent years investing in? Or do I try to take on a new hat? …it would be nice to have some models as to how someone does that.
all have day jobs. I have other businesses. It’s just not going to happen as a team. And so the question is - do I have a product? or a viable business? they are two different things.
and effort to do that. [I would need to] bring in a post-doc [to do it]… and get little, tiny grants to slowly and incrementally build this up... But that’s how you get to 14 years before it gets to practice. If the goal is really to speed these interventions into practice, then we need more support to do that.
make it difficult for us to take that next step and get it out to market. So I feel like now I know where I need to go- but I also know that I can’t necessarily do it and keep my day job… to survive in academia you need 3 grants and manage them…the time that it’s going to take to really do this start up - I would really have to leave my job.
32
a project that’s going to get me publications to get me more grants…a dissemination grant is not going to get me the number of publications, it’s not going to get me an R25 grant. It’s only going to bring in 8% of the revenue… that’s the stuff that matters
community partners. [But] If I’m doing a business I’m the one trying to find customers and paying… salespeople…and marketing people. …I’m directly on the hook for
sort of support. If we could get grants to pay for our effort towards building a startup and that was somehow legal (maybe SBIRs/STTRs work like that, I’m not sure). But if I could keep my day job and not have to do that and this on the side- which doesn’t work - that would be really great. But it’s going to take a restructuring and probably an investment from people who are interested in research actually getting out there- to buy out researchers’ time
provide support.
that would make a difference…Because then you’d have a year to focus on it.
33
fund]…and their feedback was, “We’ve never seen anything like this. You guys are unconventional - and most likely you won’t get funded.” However, he sat and he listened to us. He thought it was interesting and innovative. But he said basically that the people reviewing the grant are not familiar with this and he hadn’t seen anyone come forth with social/behavioral interventions. So I don’t think they’re ready.
And my experience is…if it is not a device - or a drug - or a patentable process- you kind of get shuffled aside to the “c-team” - not even the “b-team.” And again, I’ll say- legitimately so. Their job is to extend the value of the university and there’s really not significant financial value in the work we do - from their perspective - unless it’s
not worthwhile for the tech transfer office. They want to license technology for someone whose going to pay good money for it.
going to help us protect our intellectual property.
34
35
don’t understand the value of using the language for business even though they’re getting
potentially serve us better… Most people don’t want to start a business… these implementation models that are out there end up getting to the same place this course got us, but a little bit more behavioral research friendly.
end users if you’re working on a dissemination project versus if you’re working on a commercialization project…
what we’re doing [in this course]. D&I feels very different than what we’re doing here…this feels like sustainability and commercialization.
philosophy or orientation. We were talking about how your competitors could get at the idea first… or why did your customers go to somebody else… this is philosophically so different from our idea of “I want to support this population- and yes- I hope there are competitors who will be cheaper”…So, it’s not just the terminology, it’s also “what’s our end goal?” … It’s very different when you’re thinking about, “I’m going to build a company based on this intervention” versus “I hope my intervention gets picked up by a health care system so that we can improve outcomes.” They can have some common goals, but philosophically, they’re quite different.
36
commercialize and disseminate
engagement aspect... it feels like those are two different roles and I don’t know why they’re not
that world in-between - that is the bridge. [and now] we’re ready to work through that... Do we need to trademark a name… do we open an LLC to do our consulting... it’s been really important to shift things forward so we continue to pursue research but are also able to simultaneously move towards commercialization.
grant? How are you going to sustain yourself? it is important to consider that sustainability often times may not come from that NIH grant…we have to think of different avenues of revenue in order to continue the work that we’re trying to do.
learning the skills about how to…scale it up and how to get it out there, and maybe some [ways] we can cover our costs…Our next [step] is a big effectiveness study in the real
this in…now moving into dissemination science (which I don’t have training in), I feel much better equipped to move in that direction having this training.
www.cancer.gov www.cancer.gov/espanol