Ad Serving at Spotify Scale A journey of incremental full stack - - PowerPoint PPT Presentation

▶
ad serving at spotify scale
SMART_READER_LITE
LIVE PREVIEW

Ad Serving at Spotify Scale A journey of incremental full stack - - PowerPoint PPT Presentation

Oct 30, 2022 170 likes •1.06k views

Ad Serving at Spotify Scale A journey of incremental full stack overhaul Kinshuk Mishra, Director of Engineering kinshuk@spotify.com @_kinshukmishra A lucky mistake Expected consequences Sarcastic empathy Some valuable feedback The

slide-1
SLIDE 1

Ad Serving at Spotify Scale

A journey of incremental full stack overhaul

Kinshuk Mishra, Director of Engineering kinshuk@spotify.com @_kinshukmishra

slide-2
SLIDE 2

A lucky mistake

slide-3
SLIDE 3

Expected consequences

slide-4
SLIDE 4

Sarcastic empathy

slide-5
SLIDE 5

Some valuable feedback

slide-6
SLIDE 6

The unintended consequence

Artist engagement for exposed users went up

slide-7
SLIDE 7

The unintended consequence

Promising insights about content promotion use-case

slide-8
SLIDE 8

The unintended consequence

Confirmation that the ad server is a powerful messaging platform

slide-9
SLIDE 9

Why should you care?

slide-10
SLIDE 10

Introduction Ad technology stack Architecture Evolution

slide-11
SLIDE 11

Introduction Ad technology stack Architecture Evolution

slide-12
SLIDE 12

What I do

  • Founded ads engineering team at Spotify in 2011
  • Build all things ads engineering - team & software
  • Major focus areas :

○ Ad delivery (Backend and Web) ○ Multi-platform native ads (Client Platform) ○ Ad performance (ML and Data)

slide-13
SLIDE 13

3 noteworthy things

slide-14
SLIDE 14

Full stack refactor Evolution at scale Pragmatic choices

slide-15
SLIDE 15

100,000,000+ MAU

slide-16
SLIDE 16

50,000,000+ Subscribers

slide-17
SLIDE 17

30,000,000+ Songs

slide-18
SLIDE 18

2,000,000,000+ Playlists

slide-19
SLIDE 19

$5,000,000,000+

Revenue paid to rightsholders

slide-20
SLIDE 20

60 Markets

slide-21
SLIDE 21

Platform Ubiquity

slide-22
SLIDE 22

Freemium business model

slide-23
SLIDE 23

Ad

slide-24
SLIDE 24
slide-25
SLIDE 25
slide-26
SLIDE 26
slide-27
SLIDE 27

Introduction Ad technology stack Architecture Evolution

slide-28
SLIDE 28

Beauty of Ad Server

Relevancy Pacing Unique View Sequence Optimization

slide-29
SLIDE 29

Complexity of Ad tech ecosystem

slide-30
SLIDE 30

In essence it is pretty simple

Client User Profile database Ad Server Campaign Management Portal Billing/ Reporting Ad campaign database Data Collection System

slide-31
SLIDE 31

Spotify Ads infrastructure in 2011

Edge Service Desktop Log Delivery HDFS User Profile

Batch

Basic Ad Server Campaign Management Billing/ Reporting

slide-32
SLIDE 32

Spotify Ads infrastructure in 2017

iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP

Stream Batch

Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting

slide-33
SLIDE 33

Multi-platform clients

iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP

Stream Batch

Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting

slide-34
SLIDE 34

Data collection

iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP

Stream Batch

Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting

slide-35
SLIDE 35

Intelligence

iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP

Stream Batch

Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting

slide-36
SLIDE 36

Ad Delivery

iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP

Stream Batch

Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting

slide-37
SLIDE 37

Demand fulfillment

iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP

Stream Batch

Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting

slide-38
SLIDE 38

Now you know too

Ad server is a powerful messaging platform

slide-39
SLIDE 39

Introduction Ad technology stack Architecture Evolution

slide-40
SLIDE 40

Architecture overhaul is hard

  • While keeping the business running
  • While innovating on new products
  • When you should have done it yesterday
slide-41
SLIDE 41

Why did Spotify evolve Ads architecture?

slide-42
SLIDE 42

Future needs

  • Growth in scale
  • Emergence of new client platforms
  • Cheap cloud computing
  • New products to meet business objectives
  • Technical debt
slide-43
SLIDE 43

The 3 stories

slide-44
SLIDE 44

Fixing the legacy mess

Story 1

slide-45
SLIDE 45

Original ad server design

Edge Service Router

hash(userid)

Ad server ring with partitions

Ad server instance

Memcache Memcache Memcache Memcache Campaign DB User DB

Desktop

Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication

slide-46
SLIDE 46

Problems

slide-47
SLIDE 47

Stateful service with faulty persistence

slide-48
SLIDE 48

Cache as a data store

slide-49
SLIDE 49

Service cluster as a hashed ring

slide-50
SLIDE 50

Ad decisioning in Client

slide-51
SLIDE 51

Batch Client-Server Calls

slide-52
SLIDE 52

Fix strategy

slide-53
SLIDE 53

Fix strategy tactic

slide-54
SLIDE 54

Isolate refactor to one system at a time

slide-55
SLIDE 55

The ad server transition

Edge Service Log Delivery HDFS User Profile

Batch

Smart Ad Server Campaign Management Billing/ Reporting Ad Server Proxy (routing) Basic Ad Server

Gradual transition from basic to smart ad serving

Desktop

Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication

slide-56
SLIDE 56

After the ad server transition

Proxy Service Log Delivery HDFS User Profile

Batch

Campaign Management Billing/ Reporting Smart Ad Server Desktop

Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication

slide-57
SLIDE 57

Lean, mean and fast

Story 2

slide-58
SLIDE 58

Division of responsibilities

Desktop iOS Android Ads SDK Desktop Web Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication Ad decisioning Ad fetch

  • rchestration

Client context Ad Trigger & Render

Before After

slide-59
SLIDE 59

Problems

slide-60
SLIDE 60

Thick Clients

slide-61
SLIDE 61

Logic duplication

slide-62
SLIDE 62

Tightly coupled monolith

slide-63
SLIDE 63

Fix strategy

slide-64
SLIDE 64

Reduce State Management

slide-65
SLIDE 65

Break monolith into services

slide-66
SLIDE 66

Isolate platform independent logic into a lib

slide-67
SLIDE 67

Fix tactic

slide-68
SLIDE 68

Design your systems to be master of one thing

slide-69
SLIDE 69

Remember division of responsibilities?

Desktop iOS Android Ads SDK Desktop Web Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication Ad decisioning Ad fetch

  • rchestration

Client context Ad Trigger & Render

BAD GOOD

slide-70
SLIDE 70

Multiplatform Client design

iOS Proxy Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP

Stream Batch

Ad Server Decision Delivery Ad Exchanges Campaign Management Modeling Self-Serve Service Creative Generation Payments Billing/ Reporting

slide-71
SLIDE 71

Knowledge is power, Unreliable data is your enemy

Story 3

slide-72
SLIDE 72

Event Stream Historical ETL1 ETL2 ETL3

UserEntity1(attribute1, attribute2) UserEntity1(attribute1, attribute3) UserEntity1(attribute1, attribute3’)

slide-73
SLIDE 73
slide-74
SLIDE 74

Problems

slide-75
SLIDE 75

Duplicate, undiscoverable and fragmented datasets

slide-76
SLIDE 76

Metric inaccuracy

slide-77
SLIDE 77

Overloaded Data Infra

slide-78
SLIDE 78

Fix strategy

slide-79
SLIDE 79

Focus on reliable and timely log delivery

slide-80
SLIDE 80
slide-81
SLIDE 81

Data engineering with SLA

slide-82
SLIDE 82

Dataset canonicalization

slide-83
SLIDE 83

Some useful lessons learnt from architectural overhaul

slide-84
SLIDE 84

Test with minimal impact radius

slide-85
SLIDE 85

Mistakes are inevitable

slide-86
SLIDE 86

Speed up build decisions

slide-87
SLIDE 87

Think for tomorrow, Solve for today

slide-88
SLIDE 88

Thank You!

kinshuk@spotify.com @_kinshukmishra