Democratizing online controlled experiments at Booking.com How to - - PowerPoint PPT Presentation

▶

Nov 15, 2022 233 likes •610 views

Democratizing online controlled experiments at Booking.com How to run 1.000+ concurrent experiments Lukas Vermeer | Director of Experimentation Booking. Founded in 1996 28,000,000+ reported listings 1,550,000+ room nights per day 17.000+

SLIDE 1

Democratizing online controlled experiments at Booking.com

How to run 1.000+ concurrent experiments

Lukas Vermeer | Director of Experimentation

SLIDE 2

Founded in 1996 28,000,000+ reported listings 1,550,000+ room nights per day 17.000+ employees 1.000+ concurrent tests 2.000+ people testing

Booking.

SLIDE 3

TL;DR.

Central repository of successes and failures Descriptions of all experiment iterations and of the final decision are available for all experimenters. Genericity and extensibility Experimental design is abstracted away. Reports are automated and product agnostic. Data which can be trusted We monitor the validity of the data by computing common metrics in two separate date pipelines. Loose coupling Business logic and experiment infrastructure are purposefully kept decoupled. Building safeguards We encourage sound methodology and provide data quality checks, but no rules or automation.

SLIDE 4

Focus.

SLIDE 5

This is a story about culture

SLIDE 6

This is a story about people

SLIDE 7

1. Ask PR department for “pretty photos of

employees”.

2. Filter out non-experiment roles (dba,

support, management, etc).

3. Select six at random from the remainder.
4. Email them and ask “how do you use

experiments in your job”.

5. Copy paste replies onto slides.
6. Highlight key phrases.
7. Sprinkle in some context.

Method.

Sample Survey Results

SLIDE 8

Empower people to experience the world.

SLIDE 9

Clyde Li.

Client Side Developer

“Data-driven is quite common these days in tech industry, however, empowering everyone to make data-driven decisions independently is quite unique in Booking.com.”

SLIDE 10

Nekeia Boone.

Senior UX Copywriter

“I can come up with an idea

ver breakfast, bike to the
ffice and have it live well

before lunch. I’ve never worked anywhere else that gives me this level of

wnership and creative

freedom to validate my ideas.”

SLIDE 11

Heloisa Biagi.

Client Side Developer

“I think it's great that the company encourages everyone to hack and test ideas among the users. No higher value opinions, no centralization of decision making, everyone is free to have their own ideas and at the end of the day, users are the ones who decide what's best for them.”

SLIDE 12

You cannot have a democracy without empowering the people

SLIDE 13

Hadeer Younis.

Full-Stack Software Developer

“Experimentation is a great way to figure out if something as small as a copy change or something as big as a whole book process flow will help users, but you can’t depend on it as your only source of product validation as this will greatly hinder the product development.”

SLIDE 14

“The plural of anecdote is not data”

Lots of misinformed speakers at conferences I’ve attended

SLIDE 15

“The plural of anecdote is not data”

Raymond Wolfinger (1969)

SLIDE 16

Grey Blue Orange White 80% 60% 70% 50%

Data is just data.

To make good decisions, we need good evidence.

SLIDE 17

A B

Website Optimisation.

Let’s change the button from yellow to blue and see if it increases the magic number.

Buy now! Buy now!

SLIDE 18

A B

Hypothesis Testing.

We observed in user research that some people have difficulty finding the “buy now”

button. We suspect this is caused by the low contrast between the font and the
background. To solve this user issue, we will change the button from yellow to blue. If this

solution works, we expect to see more users hover and click, and eventually purchase.

Buy now! Buy now!

SLIDE 19

Finn Hansen.

Product Owner

“We use experimentation to help us validate hypotheses with the goal of addressing well defined user problems. It's all about learning as fast as possible in the most rigorous way possible.”

SLIDE 20

Based on [prior] we believe [condition] for [users] will encourage them to [behavior] We will know this when we see [effects] happen to [metrics] This will be good for customers, partners and our business because [motivation].

Hypothesis template.

Theory Validation Objective

SLIDE 21

9/10 tests “fail”.

“90% of product decisions have an inconclusive or negative effect on a product’s primary metric.”

VWO (2016)

90% 10%

SLIDE 22

https://medium.com/booking-com-development/moving-fast-breaking-things-and-fixing-them-as-quickly-as-possible-a6c16c5a1185

SLIDE 23

https://medium.com/booking-com-development/moving-fast-breaking-things-and-fixing-them-as-quickly-as-possible-a6c16c5a1185

SLIDE 24

Diogo Antunes.

Principal Developer & Fellow

“We work in a global scale. By validating my changes through experimentation I continuously get challenged

n my assumptions and the

way I look into the experience of our customers.”

SLIDE 25

SLIDE 26

Disclaimer: not what the actual experiment looked like.

SLIDE 27

The order of items can be improved. The order of items matters to users. The feature can be improved. The feature matters to users. The page can be improved. What is on the page matters to users.

Assumptions.