building a culture of experimentation at Spotify @bendressler - - - PowerPoint PPT Presentation

▶
building a culture of experimentation at spotify
SMART_READER_LITE
LIVE PREVIEW

building a culture of experimentation at Spotify @bendressler - - - PowerPoint PPT Presentation

Feb 21, 2023 268 likes •1.03k views

building a culture of experimentation at Spotify @bendressler - experimentation lead user retention product satisfaction # of track plays predicting things is hard hi expectations why and how we experiment - in a consumer oriented live

slide-1
SLIDE 1
slide-2
SLIDE 2

building a culture of experimentation at Spotify

@bendressler - experimentation lead

slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5

user retention

slide-6
SLIDE 6

product satisfaction

slide-7
SLIDE 7

# of track plays

slide-8
SLIDE 8

predicting things is hard

slide-9
SLIDE 9
slide-10
SLIDE 10

hi

slide-11
SLIDE 11

expectations

why and how we experiment - in a consumer oriented live product where we aim to drive a defined set of KPIs.

slide-12
SLIDE 12

expectations

building and evaluating a/b testing basics graduating towards experimentation showcase

slide-13
SLIDE 13

prologue

slide-14
SLIDE 14

“and God said, ‘Let there be light,’ and there was light” genesis

building

slide-15
SLIDE 15

“God saw all that he had made. and it was very good” genesis

evaluating

slide-16
SLIDE 16

a successful idea does what it’s supposed to do

slide-17
SLIDE 17

ALMIGHTY BUSINESS KPI =

slide-18
SLIDE 18

All my favorites are there I use it on the subway I don’t get it ITS AMAZING It doesn’t support Linux, I’m out

ALMIGHTY BUSINESS KPI =

slide-19
SLIDE 19

X 100,000,000

All my favorites are there I use it on the subway I don’t get it ITS AMAZING It doesn’t support Linux, I’m out

ALMIGHTY BUSINESS KPI =

slide-20
SLIDE 20

I don’t get it

slide-21
SLIDE 21

struggles with finding his friend’s profile used to be an iTunes user “what’s the difference between saving and adding to a playlist?” doesn’t know there is a Premium version

slide-22
SLIDE 22

11% 51% 91% feeling ‘super pumped’

slide-23
SLIDE 23

so what else do you want?

slide-24
SLIDE 24

problem #1

slide-25
SLIDE 25

“omg, people who build tons

  • f playlists stay with us

forever!” “let’s call the CEO!!” “mandatory playlist creation for everyone!!!”

#hypotheticalscenario

slide-26
SLIDE 26

playlisting retention

slide-27
SLIDE 27

playlisting retention

slide-28
SLIDE 28

music enthusiasm playlisting retention tech expertise

slide-29
SLIDE 29

problem #2

slide-30
SLIDE 30
slide-31
SLIDE 31

enter: science

slide-32
SLIDE 32

“I wonder why some people end up being super mean to others”

slide-33
SLIDE 33

education personality genetics ethnicity experiences ideology socio-economic status situational factors

?

slide-34
SLIDE 34

education personality genetics ethnicity experiences ideology socio-economic status situational factors

?

slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37

“now punch those guys over there”

slide-38
SLIDE 38

education personality genetics ethnicity experiences ideology socio-economic status situational factors

!

slide-39
SLIDE 39

a/b testing & experimentation

slide-40
SLIDE 40

b

(control + change)

a

(control)

  • bservation a

population

  • bservation b

difference = effect

slide-41
SLIDE 41

b

(control + change)

a

(control)

  • bservation a

population

  • bservation b

difference = effect

enough participants random assignment

slide-42
SLIDE 42

b

(control + change)

a

(control)

  • bservation a

population

  • bservation b

difference = effect

make changing things cheap & easy, QA

slide-43
SLIDE 43

b

(control + change)

a

(control)

  • bservation a

population

  • bservation b

difference = effect

reliable data, good logging, meaningful metrics

slide-44
SLIDE 44

b

(control + change)

a

(control)

  • bservation a

population

  • bservation b

difference = effect

statistical test

slide-45
SLIDE 45

guarantee statistical assumptions minimise delay from start to finish handle many concurrent tests minimise human error

slide-46
SLIDE 46

“God looked at his conversion rate. and it was very good”

slide-47
SLIDE 47

large buttons are swee-heet blue links, no, underlined, no, red links flashing text gets more attention

  • kittens. just kittens.
slide-48
SLIDE 48

“A/B testing is inevitably reductive, darwinistically evolving the ‘fittest’ design ... it forces you to follow your audience, not lead them ... but it is best suited to niche-testing elements, not layouts.”

Martin Gittins

slide-49
SLIDE 49

“an experiment is a means of gathering information to compare an idea against reality”.

from Experiment! by Colin McFarland

slide-50
SLIDE 50

“God said ‘let our value proposition be clearer to new users’”

slide-51
SLIDE 51

+ strong hypothesis + meaningful metrics + community

slide-52
SLIDE 52

strong hypothesis

  • what. why. how. who.
slide-53
SLIDE 53

meaningful metrics

  • who. when. where. what.

relevant and useful.

slide-54
SLIDE 54

community

  • critique. share. repeat.
slide-55
SLIDE 55

in practice

slide-56
SLIDE 56
slide-57
SLIDE 57
slide-58
SLIDE 58
slide-59
SLIDE 59
slide-60
SLIDE 60

example

slide-61
SLIDE 61

“God said ‘let our value proposition be clearer to new users’”

slide-62
SLIDE 62
slide-63
SLIDE 63
slide-64
SLIDE 64
slide-65
SLIDE 65
slide-66
SLIDE 66
slide-67
SLIDE 67
slide-68
SLIDE 68
slide-69
SLIDE 69

direction

slide-70
SLIDE 70

epilogue

slide-71
SLIDE 71

“and God said, ‘Let there be light,’”

building

slide-72
SLIDE 72

“and God said, ‘Let there be light,’ - so the angels asked what his hypothesis was.”

building

slide-73
SLIDE 73

thanks :)

@bendressler (twitter, medium, email)

slide-74
SLIDE 74

further reading

microsoft (ronny kohavi) booking.com (lukas vermeer, erin weigel) skyscanner (colin mcfarland) etsy (dan mckinley) linkedin (ya xu) riot games (jeffrey lin)

slide-75
SLIDE 75