SLIDE 1
How We Know Where You Are in House of Cards @zimmermatt Netflix - - PowerPoint PPT Presentation
How We Know Where You Are in House of Cards @zimmermatt Netflix - - PowerPoint PPT Presentation
Netflixs Viewing Data Microservices How We Know Where You Are in House of Cards @zimmermatt Netflix Scale @zimmermatt Netflix Scale 62,000,000 @zimmermatt Netflix Scale 62,000,000 50+ @zimmermatt Netflix Scale 62,000,000 50+
SLIDE 2
SLIDE 3
SLIDE 4
Netflix Scale
@zimmermatt
SLIDE 5
Netflix Scale
62,000,000
@zimmermatt
SLIDE 6
Netflix Scale
62,000,000 50+
@zimmermatt
SLIDE 7
Netflix Scale
62,000,000 50+ 3,000,000,000+
@zimmermatt
SLIDE 8
Netflix Scale
62,000,000 50+ 3,000,000,000+ 1000+
@zimmermatt
SLIDE 9
Netflix Scale
62,000,000 50+ 3,000,000,000+ 1000+ 37%
@zimmermatt
SLIDE 10
Viewing Data
@zimmermatt
SLIDE 11
Viewing Data: History
Who
@zimmermatt
SLIDE 12
Viewing Data: History
Who, What
@zimmermatt
SLIDE 13
Viewing Data: History
Who, What, When
6/10/2015 11:55
@zimmermatt
SLIDE 14
Viewing Data: History
Who, What, When, Where
6/10/2015 11:55
@zimmermatt
SLIDE 15
Viewing Data: History
Who, What, When, Where, How Long
6/10/2015 11:55
@zimmermatt
SLIDE 16
Viewing Data: Telemetry
@zimmermatt
SLIDE 17
What have I watched?
@zimmermatt
SLIDE 18
What have I watched?
@zimmermatt
SLIDE 19
Where was I at?
@zimmermatt
SLIDE 20
What else am I watching?
@zimmermatt
SLIDE 21
What Happened?
Video Bitrate Usage
@zimmermatt
SLIDE 22
SLIDE 23
Viewing Data Requests
@zimmermatt
SLIDE 24
Viewing Data Requests
@zimmermatt
SLIDE 25
Viewing Data Requests
@zimmermatt
SLIDE 26
Viewing Data Requests
@zimmermatt
SLIDE 27
Viewing Data Requests
@zimmermatt
SLIDE 28
SLIDE 29
Gen 3 (Existing) System
Summarized Sessions Viewing History History Cache Logs
@zimmermatt
Viewing Service Stateful Tier
1 n-2 n-1 …
Active Sessions Latest Positions Session Summary
Stateless Tier (History, logs, and fallback)
Events
SLIDE 30
Gen 3 System’s Architecture & Scale
Logs
@zimmermatt
SLIDE 31
Gen 3 System’s Architecture & Scale
Viewing History Logs
@zimmermatt
SLIDE 32
Gen 3 System’s Architecture & Scale
Viewing History History Cache Logs
@zimmermatt
SLIDE 33
Gen 3 System’s Architecture & Scale
Viewing History History Cache Logs Summarized Sessions
@zimmermatt
SLIDE 34
Gen 3 System’s Architecture & Scale
Viewing History History Cache
Viewing Service Stateful Tier
… Stateless Tier (History, logs, and fallback) Logs Summarized Sessions
@zimmermatt
SLIDE 35
Gen 3 System’s Architecture & Scale
Viewing History History Cache
Viewing Service Stateful Tier
1 n-2 n-1 …
Active Sessions Latest Positions Session Summary
Stateless Tier (History, logs, and fallback)
Events
Logs Summarized Sessions
@zimmermatt
SLIDE 36
SLIDE 37
Growth
2015 @zimmermatt
SLIDE 38
Virtuous Cycle
Improved Personalization Better Experience Viewing
@zimmermatt
SLIDE 39
Stateful Instance Count Over Time
@zimmermatt
SLIDE 40
Guiding Lights
SLIDE 41
SLIDE 42
SLIDE 43
SLIDE 44
Externalize State…
Viewing History History Cache
Viewing Service Stateful Tier
1 n-2 n-1 …
Active Sessions Latest Positions Session Summary
Stateless Tier (History, logs, and fallback)
Events
Logs Summarized Sessions
@zimmermatt
SLIDE 45
Externalize State… DENIED!
Viewing History History Cache
Viewing Service Stateful Tier
1 n-2 n-1 …
Active Sessions Latest Positions Session Summary
Stateless Tier (History, logs, and fallback)
Events
Logs Summarized Sessions
@zimmermatt
SLIDE 46
Service Decomposition
Video Bitrate Usage
@zimmermatt
SLIDE 47
Service Decomposition
Start Stop @zimmermatt
SLIDE 48
Active Sessions Last Position Viewing History Data Feed
Service Decomposition
Start Stop @zimmermatt
SLIDE 49
Active Sessions Last Position Viewing History Data Feed
Service Decomposition
Start Stop Event Stream Stream State Session Summary @zimmermatt
SLIDE 50
Service Decomposition
Collect Process Provide
Events Queries
@zimmermatt
SLIDE 51
Gen 3 Review
Viewing History History Cache Logs Summarized Sessions
@zimmermatt
Viewing Service Stateful Tier
1 n-2 n-1 …
Active Sessions Latest Positions Session Summary
Stateless Tier (History, logs, and fallback)
Events
SLIDE 52
Gen 4 Generic Architecture
Processor Provider Collector
@zimmermatt
SLIDE 53
Gen 4 Generic Architecture
Processor Provider Collector
Queue
@zimmermatt
SLIDE 54
Gen 4 Generic Architecture
Processor Provider Collector
Materialized Views Raw Data Queue
@zimmermatt
SLIDE 55
Gen 4 Generic Architecture
Processor Provider Collector
Materialized Views Cache Raw Data Queue
@zimmermatt
SLIDE 56
Gen 4 Architecture
Session Summary
Processor
Summarized Sessions
Materialized View
Session Events
Collector
Event Data
Raw Data
Log
Collector
Logs
Raw Data
Viewing History Positions Active Sessions
Materialized View
Session Accounting
Provider
Playback History
Provider
History
Raw Data
@zimmermatt
SLIDE 57
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Shadow Testing
Edge Service
Summarized Sessions
Viewing History
History Cache Logs Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Log Collector Logs Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
SLIDE 58
Shadow Testing: Start Play Event
Edge Service
Session Events Collector
Event Data
Active Sessions Session Accounting Provider
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 59
Shadow Testing: Start Play Event
Edge Service
Session Events Collector
Event Data
Active Sessions Session Accounting Provider
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 60
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Shadow Testing: Start Play Event
Edge Service
Session Events Collector
Event Data
Active Sessions Session Accounting Provider
@zimmermatt
SLIDE 61
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Shadow Testing: Start Play Event
Edge Service
Session Events Collector
Event Data
Active Sessions Session Accounting Provider
@zimmermatt
SLIDE 62
Shadow Testing: Start Play Event
Edge Service
Session Events Collector
Event Data
Active Sessions Session Accounting Provider
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 63
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events Viewing History Positions Playback History Provider History
Shadow Testing: Read Latest Position
Edge Service @zimmermatt
SLIDE 64
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Shadow Testing: Read Latest Position
Edge Service
Viewing History Positions Playback History Provider History
@zimmermatt
SLIDE 65
Shadow Testing: Read Latest Position
Edge Service
Viewing History Positions Playback History Provider History
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 66
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
SLIDE 67
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
SLIDE 68
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
SLIDE 69
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
SLIDE 70
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 71
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 72
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 73
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 74
Shadow Testing: Stop Event
Edge Service
Summarized Sessions Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 75
Shadow Testing: Stop Event
Summarized Sessions Session Summary Processor Summarized Sessions
@zimmermatt
SLIDE 76
Shadow Testing
Edge Service
Summarized Sessions
Viewing History
History Cache Logs Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Log Collector Logs Active Sessions Session Accounting Provider Viewing History Positions Playback History Provider History
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 77
Traffic Dial
Session Summary Processor Summarized Sessions Session Events Collector
Event Data
Log Collector Logs Active Sessions Session Accounting Provider
Edge Service
Viewing History Positions Playback History Provider History Summarized Sessions
Viewing History
History Cache Logs
@zimmermatt
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
SLIDE 78
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Traffic Dial
Session Events Collector
Event Data
Log Collector Active Sessions Session Accounting Provider
Edge Service
Positions Playback History Provider Summarized Sessions
Viewing History
History Cache Logs Session Summary Processor
@zimmermatt
SLIDE 79
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Traffic Dial
Session Events Collector
Event Data
Log Collector Active Sessions Session Accounting Provider
Edge Service
Positions Playback History Provider Summarized Sessions
Viewing History
History Cache Logs Session Summary Processor
@zimmermatt
SLIDE 80
Viewing Service Stateful Tier 1 n-2 n-1
…
Active Sessions Latest Positions Session Summary Stateless Tier (History, logs, and fallback) Events
Traffic Dial
Session Events Collector
Event Data
Edge Service
Summarized Sessions
Viewing History
History Cache Logs Session Summary Processor
@zimmermatt
Active Sessions Session Accounting Provider Positions Playback History Provider
SLIDE 81
Viewing Service Stateful Tier 1 n-2 n-1
…
Latest Positions Stateless Tier (History, logs, and fallback)
Traffic Dial
Edge Service
Viewing History
History Cache
@zimmermatt
Positions Playback History Provider
SLIDE 82
Viewing Service Stateful Tier 1 n-2 n-1
…
Latest Positions Stateless Tier (History, logs, and fallback)
Traffic Dial
Log Collector
Edge Service
Logs
@zimmermatt
SLIDE 83
Viewing Service Stateful Tier 1 n-2 n-1
…
Latest Positions Stateless Tier (History, logs, and fallback)
Traffic Dial
Edge Service
Playback History Provider
Viewing History
History Cache
@zimmermatt
SLIDE 84
15
SLIDE 85
Migration Status
Session Events Collector
Event Data
Active Sessions Session Accounting Provider
Edge Service
Session Summary Processor Summarized Sessions Log Collector Logs Viewing History Positions Playback History Provider History
@zimmermatt
SLIDE 86
Remember this?
@zimmermatt
SLIDE 87
This is a typical graph now…
@zimmermatt
SLIDE 88
The Road Ahead
@zimmermatt
SLIDE 89
Key Points
- Devour the whale a bite at a time.
- Use CQRS / Event Sourcing. Backup the raw event
- data. Design for idempotency so it can be replayed.
- System architectures are throw-away artifacts, useful for
- nly a limited time.
@zimmermatt
SLIDE 90