Validation and Experimentation Board of Innovation The Growth - - PowerPoint PPT Presentation

Board of Innovation 
 The Growth Revolution

Validation and Experimentation

Intro Validation of new Ventures Experimentation Process Assumptions Hypotheses Experiment Types Metrics Wrap-up

Introductions

HI THERE!

• 3

ABOUT US

BOARD OF INNOVATION

• 4

• 5

We help corporates world wide to design & execute innovation programs.

ALL INDUSTRIES B2B/B2C

• 6

Thinking global

It crosses borders and countries. Our innovation programs have been delivered in any geographical scope and cultural context.

HQ based in Antwerp (Belgium)

22

Countries

5

Continents

10+

Nationalities

A TEAM CROSSING BORDERS

We help corporates world wide to design & execute innovation

• programs. We look forward to

select & tailor our services according to your needs.

Business Design

Scoping workshop Strategy Fit User Research Customer Journey Mapping Opportunity Scan Problem Fit Innovation Coach Ideation Brainstorm Co-creation workshop Validation as a service Business Model Design Experiment design Solution Fit Entrepreneur in residence Venture Team Coaching Market Fit Design Sprints Innovation accelerator External startup accelerator Special Formats

Transformation

Growth & Innovation Strategy Innovation Program Design Building Capabilities Culture Program

Talent Development

Train-the-trainer E-courses Toolkit development Innovation At scale

+1000 people

Leadership Young graduates Innovation manager coaching Talent Programs

± 6 months

Innovation for executives Strategy for designers Innovation Strategy Design Thinking Innovation & Creativity Intrapreneurship/Lean startup Business Model Innovation Training

1-2 days

Keynotes Innovation Day Inspiration

1 day

Our services

• 7

• 8

TARRYN GILLES

What?

• 10

What is validation?

validation • n, the process of gathering evidence and learnings around business ideas through experimentation and user testing, in order to make faster, informed, de-risked decisions.

Looking to scale a business idea?

• 11

Startups are good at finding out what to build. Corporates are good at scaling. With Validation as a Service, we’ll merge the two approaches to help corporates identify and validate unmet needs in the market (problem validation), and to help design, prototype and test solutions (solution validation). Find new, quick, efficient ways to spot, validate and scale up new business opportunities.

Why?

• 13

Evidence-based decision making in a corporate context

HOW DO YOU KNOW?

• 14

Validated Learning?

Waterfall = risky

CC @lfittl

Don’t: 2 years of research before testing the market.

Release!

Risk Time

build build build build

• 15

Validated Learning? Change Mindset!

Lean = shorter cycles!

Release!

Risk Time

Release! Release! Release! build build build build

Focus on experiments & validation

• Test critical assumptions
• Minimise risk by maximal learning.

💢 Fail and learn fast

How?

• 17

In which stage are you?

Depending on the stage of the innovation funnel you’re in, you’ll need to validate different elements of your business idea, so that you can pick different kinds of experiments.

• Problems: at the very

beginning of your innova2on path, you’ll need to test whether a problem you iden2fied is a problem worth solving for your customer.


• Solu0ons: Does your offer

solve this need and is the customer willing to pay?


• Features: test core

features that are crucial for adding value to your solu2on.


• Business Model: test the

viability of the solu2on you designed.
 
 


• Pricing: test the pricing

model of your product or

• service. 


• 18

Cleary define your challenge, and at what stage of the innovation funnel you’re in, in

• rder to select relevant

experiment types to validate various aspects of your business proposition.

• A. Define your

focus: what do you seek to validate?

Validation step by step

• 19

Cleary define your challenge, and at what stage of the innovation funnel you’re in, in

• rder to select relevant

experiment types to validate various aspects of your business proposition.

• A. Define your

focus: what do you seek to validate?

• B. Map and

prioritise your critical assumptions

Validation step by step

Regardless the stage you’re in (problem space, solution space, business or pricing models), you can now map

• ut the assumptions and then

turn them into hypotheses to be tested.

• 20

Cleary define your challenge, and at what stage of the innovation funnel you’re in, in

• rder to select relevant

experiment types to validate various aspects of your business proposition.

• A. Define your

focus: what do you seek to validate?

• B. Map and

prioritise your critical assumptions

Validation step by step

Regardless the stage you’re in (problem space, solution space, business or pricing models), you can now map

• ut the assumptions and then

turn them into hypotheses to be tested.

• C. Design the relevant

experiments to test your hypotheses

Once you have ranked your assumptions and defined key hypotheses you want to test, you can select the most suitable experiment(s) with the right metrics to test and validate (or reject) your hypotheses.

• 21

Cleary define your challenge, and at what stage of the innovation funnel you’re in, in

• rder to select relevant

experiment types to validate various aspects of your business proposition.

• A. Define your

focus: what do you seek to validate?

• B. Map and

prioritise your critical assumptions

Validation step by step

Regardless the stage you’re in (problem space, solution space, business or pricing models), you can now map

• ut the assumptions and then

turn them into hypotheses to be tested.

• C. Design the relevant

experiments to test your hypotheses

Once you have ranked your assumptions and defined key hypotheses you want to test, you can select the most suitable experiment(s) with the right metrics to test and validate (or reject) your hypotheses.

• D. Interpreting results

and making (tough) evidence-based decisions.

Deciding whether to pivot, stop or persevere or iterate based on your learnings is a crucial moment and

• ften the toughest part - what is the

evidence telling me? What do I do next? Is this enough?

Validation of New Business Ideas

• 22

2001: “IT”

• 24

BONO JEFF BEZOS

• 25

‘’IT’’ would be “Bigger than the internet” “As big a deal as the PC!” “A Transforming factor for cities”

• 26

“IT” turned out to be …

A TWO-WHEELED, SELF-BALANCING, ELECTRIC PERSONAL TRANSPORTATION DEVICE , USING PATENTED DYNAMIC STABILIZATION TECHNOLOGY FOR ALL PEOPLE THAT TRAVEL SHORT DISTANCES ….

“If enough people see this machine, you won’t have to convince them to architect cities around it; it’ll just happen.” Steve Jobs

• 28

Lessons learned?

Consider your assumptions Validate the key problem FOCUS Validate your solution Validate your business model Validate your pricing structure

BEFORE BUILDING AND LAUNCHING

“Life is too short to build something nobody wants”

• Ash Maurya, Running Lean

• 30

Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day. Jeff Bezos

• 31

Launching new business ideas

MARKET FIT PROBLEM FIT SOLUTION FIT GROWTH & SCALE DISCOVERY EXPERIMENTATION

• 32

DO’s DONT’s

Choosing your focus - what are you aiming to validate? Take on the right mindset! See this as a way to learn! “Move fast, with stable infrastructure” Don’t skip the discovery phase - it is crucial for the groundwork. Get attached to your solution(s). Just don’t. OK? Don’t be afraid of imperfections or embarrassment.

• 33

Don’t ask “what will we build?”

Ask “what do you want to learn?”

The Experimentation Process

• 34

• 35

Experimentation is the process of

• bserving how customers react to

validate (or invalidate) business idea assumptions.

Assumptions

CUSTOMER

DESIGN YOUR EXPERIMENT RUN EXPERIMENT ANALYZE RESULTS TAKE AN EVIDENCE-BASED DECISION IDENTIFY YOUR RISKIEST ASSUMPTION

The Experiment Process

• 37

The Lean Startup

Measure Build (MVP)

Learn

Learn

Learn from your failures And iterate!

Assumptions

CUSTOMER

DESIGN YOUR EXPERIMENT RUN EXPERIMENT ANALYZE RESULTS TAKE AN EVIDENCE-BASED DECISION

IDENTIFY YOUR RISKIEST ASSUMPTION

The Experiment Process

First, know that you don’t know

• 40

ASSUMPTIONS REALITY

WHAT YOU THINK IS TRUE WHAT YOU KNOW IS TRUE

EXPERIMENTATION

• 41

Mapping out the assumptions

Divide the assumptions into categories - it will help you focus on the assumptions that really matter

Desirability Client/user focus Viability Business Model Feasibility Tech constraints Corporate fit Strategy

/ Team

Assumption mapper

Get out of the building and test this NOW Start preparing you’ll need to test this as soon as possible Easy but non-essential Don’t waste your time on this

DIFFICULT TO TEST

We need time and resources to test this

NON- IMPORTANT ASSUMPTION

If this assumption is not true the concept could still survive

IMPORTANT ASSUMPTION

If this assumption is not true the concept might fail It hurts, 
 take it away

EASY TO TEST

We can test it relatively quickly

People prefer 
 black mobility vehicles People will use the vehicle for every trip they take to the grocery store Architects are willing to redesign infrastructure to accommodate Segways People prefer personal mobility vehicles for short distances

Tip💢
 Prioritize the validation of assumptions around user needs (desirability)

In the case of Segway

/ Team

Get out of the building and test this NOW Start preparing you’ll need to test this as soon as possible Easy but non-essential Don’t waste your time on this

DIFFICULT TO TEST

We need time and resources to test this

NON- IMPORTANT ASSUMPTION

If this assumption is not true the concept could still survive

IMPORTANT ASSUMPTION

If this assumption is not true the concept might fail It hurts, 
 take it away

EASY TO TEST

We can test it relatively quickly

Dot voting on most critical assumptions

Assumption mapper

• 44

If you have an assumption, you can either:

ACCEPT THE RISK CONVERT IT INTO A TESTABLE HYPOTHESIS

• r

Assumptions

CUSTOMER

DESIGN YOUR EXPERIMENT

RUN EXPERIMENT ANALYZE RESULTS TAKE AN EVIDENCE-BASED DECISION IDENTIFY YOUR RISKIEST ASSUMPTION

The Experiment Process

Hypotheses

STEP ONE OF DESIGNING YOUR EXPERIMENT

• 46

• 47

Assumption Hypothesis

An assumption is a statement that we believe to be true, without any evidence to back it up. A hypothesis is an educated guess for what we expect to happen in a given experiment. It’s a prediction we can test.

If we do X, then Y% of target audience will behave in way Z.

STIMULUS CREATED BY EXPERIMENT METRIC MEASURED (% OR #) CUSTOMER RESPONSE TO STIMULUS

Specific Repeatable Action + Expected Measurable Reaction

What makes a good hypothesis?

Are there vague words like “some people” or “customer”? Be specific. Make sure you have a well defined customer persona. Is it falsifiable? What evidence would convince a reasonable person that the hypothesis is wrong? Create a measurable hypothesis. Eliminate hedging words like “maybe,” “better,” “some” and convert to and IF ________ THEN ________ statement. Is it actually worth testing? What you would change if the experiment succeeds or fails. If you’re not going to change anything, the experiment is not worth running. Did you get a second

• pinion?

We all have blind spots. Check your work with a peer and ask them to tighten up the hypothesis.

WE BELIEVE THAT TO VERIFY THAT, WE WILL AND MEASURE WE ARE RIGHT IF

• 50

Designing Experiments

Value Proposition Validation

What do we believe? What is our assumption? What will we do to verify/ test that? What metric will we use to measure results? What are our success criteria? How do we know if we’re right?

• 51

Designing Experiments

Value Proposition Validation

WE BELIEVE THAT TO VERIFY THAT, WE WILL AND MEASURE WE ARE RIGHT IF 50+ women are interested in the value proposition of an indoor air quality monitor. Run a Facebook Ad to this target audience for a product landing page where people can click ‘Buy Now’ and then leave their email if they want to be kept updated. If at least 8% of page visitors leaves their email address . The amount of email opt-ins.

example

Experiment types

READY TO GO, BUT NO IDEA WHERE TO START?

• 52

boardofinnovation.com/validation

• 54

How to pick the right experiment

Pros and Cons Advantages and limitations of each experiment. Tools Some ready-to-use services you will find helpful to get your experimentation started

Currency These are the metrics you’ll need to measure in order to validate (or reject) your hypothesis. These results will reflect the interest/commitment of your addressed market.

Kind of experiment Evaluative: the experiment helps you evaluate a measurable hypothesis Generative: the experiment helps you gather additional insights and signals Perfect to test… Different hypotheses require different experiments. For each experiment, we specify whether it is good or not to test a specific component. Target audience While most of the experiments can be used in any industry, some can be only (or more easily) applied to a B2C context.

wireframes mockups wizard of oz concierge live product
 landing page (smoke test) high hurdle ad campaign (trigger test) a/b testing

QUALITATIVE QUANTITATIVE OFFLINE ONLINE

Types of experiments

wireframes mockups wizard of oz concierge live product
 marketing materials landing page (smoke test) explainer video ad campaign (trigger test) a/b testing

QUALITATIVE QUANTITATIVE OFFLINE ONLINE

WHY WHAT

validate behaviour explain behaviour

Types of experiments

• 58

MVP (Minimum Viable Product)

A PRODUCT WITH JUST ENOUGH FEATURES TO SATISFY EARLY CUSTOMERS AND TO PROVIDE FEEDBACK FOR FUTURE DEVELOPMENT.

Just the core feature(s) needed to realise your value proposition!

• 59

PROBABLY MORE MINIMAL THAN YOU THINK!

Eric Ries, author of the ‘The Lean Startup’

How minimal should your MVP be?

• 60

Facebook

• 61

Fake it…

• 62

MAXIMIZE LEARNINGS BY MINIMIZING THE TIME TO TEST THINGS

Tom Chi, co-founder of Google X

• 64

Easy Wireframing

• 65

Imposter Judo

Use related website/product as if it were your own. Repackage as existing product. Why build something if it already exists?

• 66

Concierge

Test manually, automate once validated

Wizard Of Oz

• 67

boring surveys HIGH HURDLE ads and audiences TRIGGER TEST landing pages SMOKE TEST A versus B SPLIT TEST

Online Experiments

Testing 2, 3, 4 versions simultaneously. One difference per version. Landing page or other form of value proposition that lets you collect leads and measure demand or compare audiences. Testing ads for copy, form, image… Which

• ne gets the best

click-through rate? Make people pay with their time.

• 68

Split Test

YOU CAN A/B TEST EVERYTHING!

• 69

Where there’s fire there’s smoke

Smoke Test

LANDING PAGE

• 70

EXPERIMENT =

a procedure carried out under controlled conditions in order to discover an unknown effect or test a hypothesis.

POSSIBILITIES = INFINITE

SPECIFIC REPEATABLE ACTION + EXPECTED MEASURABLE ACTION

• 71

1 Experiments are never standalone, they’re additive 2
 Expected

• utcomes

need to be declared up- front 3 Expected

• utcomes

need to be falsifiable 4
 Experiments need to be time-boxed 5
 Break- through ideas are hidden within “failed” experiments

Ground Rules

• 72

There are no failed experiments.
 Only unexpected outcomes.

Assumptions

CUSTOMER

DESIGN YOUR EXPERIMENT RUN EXPERIMENT

ANALYZE RESULTS TAKE AN EVIDENCE-BASED DECISION

IDENTIFY YOUR RISKIEST ASSUMPTION

The Experiment Process

• 74

Metrics help you measure progress towards your goals

WHAT MAKES A GOOD METRIC?

COMPARATIVE RATE/RATIO

If you’re busy explaining the data, you won’t be busy acting on it.

UNDERSTABLE BEHAVIOR CHANGING

Comparison is context. The only way to measure change is through tension between two metrics. What will you do differently based on the results you collect?

QUALITATIVE QUANTITATIVE

stories vs opinions, revealing, hard to aggregate, depth of insights, time consuming. numbers and stats, hard facts, less insights, easier to analyze; often sour and disappointing.

the why the what

Ways to look at data for decision-making

• 77

QUALITATIVE DATA QUANTITATIVE DATA

ARE YOUR ASSUMPTIONS (UN)TRUE? WHY ARE YOUR ASSUMPTIONS (UN)TRUE?

78

EXPLORATORY REPORTING

speculative, try to find interesting and unexpected insights, source of unfair advantages. predictable, keeps you in touch with day-to-day

• perations.

cool. necessary.

Ways to look at data for decision-making

• 79

80

CORRELATION CAUSATION

two variables that are related - but may be dependent

• n something else.

independent variable that directly impact a dependent one.

ice cream & drowning. summer time & drowning.

Ways to look at data for decision making

AIRBNB

HYPOTHESIS (GUT FEELING) if airbnb listings have professional-looking photographs more users will book them EXPERIMENT 20 field photographers posing as airbnb employees MEASURE RESULTS compare photographed listing to control group DECISION? launch photography as service for all hosts

SRSLY.

• 82

WITHOUT DATA VENTURE A GOOD GUESS WITH DATA FIND A COMMONALITY

“Gee, the houses that do well look really nice!” “Computer: what do the most booked houses have in common?” Maybe it’s the camera. Camera model.

• 83

Experiment Design in a nutshell

FIND CORRELATION TEST CAUSALITY IDENTIFY ASSUMPTION

• 84

Key Questions!

WE BELIEVE THAT TO VERIFY THAT, WE WILL AND MEASURE WE ARE RIGHT IF What do we believe? What is our assumption? What will we do to verify/ test that? What metric will we use to measure results? What are our success criteria? How do we know if we’re right?

Designing Experiments

Assumptions

CUSTOMER

DESIGN YOUR EXPERIMENT RUN EXPERIMENT ANALYZE RESULTS TAKE AN EVIDENCE-BASED DECISION IDENTIFY YOUR RISKIEST ASSUMPTION

Validation and experimentation

Hire our team of validators

We help innova,on teams in corporates to validate new business proposi,ons, to prototype, and to launch services that customers

Valida,on as a Service