Presenters Phillipa Avery Robert Reta Senior Software Engineer - - PowerPoint PPT Presentation

Presenters

Phillipa Avery

Senior Software Engineer Project Technical Lead and Engineer Downloads License Accounting pavery@Netflix.com / @PhillipaAvery

Robert Reta

Senior Software Engineer Event Sourcing System Architect Downloads License Accounting rreta@netflix.com / @rreta04

Phillipa Robert

Overview

• 1. Why do we need a Downloads License Accounting

Service?

• 2. Event Sourcing overview.
• 3. Deep dive into the Event Sourcing Architecture.
• 4. What’s it like working with the License Accounting

Service after release?

LICENSE CREATE PLAYBACK CONTEXT SESSION EVENTS (START, PAUSE, RESUME, KEEPALIVE)

Streaming Playback Lifecycle

RELEASE LICENSE SESSION EVENT (STOP)

STOP

Streaming Playback Lifecycle

LICENSE CREATE PLAYBACK CONTEXT DOWNLOADED ENCODED MOVIE

Downloads Playback Lifecycle

~ 1 year

SESSION EVENTS (START, PAUSE, RESUME, KEEPALIVE, STOP) RENEW LICENSE?

X

RELEASE LICENSE

Downloads Playback Lifecycle

Download Business Requirements

• Devices with downloads

Download Business Requirements

• Devices with downloads
• Downloads per studio

Download Business Requirements

• Devices with downloads
• Downloads per studio
• Movie downloads or playback over year

~ 1 year

License Accounting Service Requirements

❏ Flexible ❏ Debuggable ❏ Reliable ❏ Scalable

F l e x i b i l i t y Debuggability S c a l a b i l i t y R e l i a b i l i t y

License Accounting Service Requirements...

✓ Flexible: data model can be changed

❏ RDBMS ❏ Document Model

License Accounting Service Requirements...

✓ Flexible: data model can be changed

❏ RDBMS ❏ Document Model

✓ Debuggable

❏ Event Sourcing

License Accounting Service Requirements...

✓ Flexible: data model can be changed

❏ RDBMS ❏ Document Model

✓ Debuggable

❏ Event Sourcing ✓ Reliable ❏ Fallbacks

License Accounting Service Requirements...

✓ Flexible: data model can be changed

❏ RDBMS ❏ Document Model

✓ Debuggable

❏ Event Sourcing ✓ Reliable ❏ Fallbacks ✓ Scalable

Event Sourcing

Domain Model

Make Purple Command

Commands

Commands

Make Purple Command

Command Handler

Added Red Event Added Blue Event Make Purple Command

Command Handler

Commands

Events

Added Red Event Added Blue Event

Event Handler

Events

Added Blue Event

Event Handler

Domain Model

Aggregate

Commands

Make Red Command

Command Handler

Command Handler

Commands

Make Red Command

Aggregate Service

REST Endpoint

Aggregate Repository

Event Store

Query Query Query

Event Store

Row ID Events

Event Store

Row ID Events

Row ID Events

e1

e2 e3 e5 e6 e7 e4

Event Store

Aggregate id 1 Aggregate id 2

Aggregate Repository

Query

Event Store

Aggregate Repository Event Store

Query

select * from events where rowId =

Aggregate Repository Event Store

Response

e1

e2 e3 e5 e6 e7 e4

Aggregate Repository

e1

e2 e3 e5 e6 e7 e4

Aggregate id 1 Aggregate id 2

e1

e2 e5 e6 e3 e4 e7

Aggregate Repository

Aggregate id 2 Aggregate id 1

e1

e2 e5 e6 e3 e4 e7 Aggregate 1 Aggregate 2

Aggregate Repository

Aggregate 2 Aggregate 1

Aggregate id 2 Aggregate id 1

e1

e2 e5 e6 e3 e4 e7

Aggregate Repository

Aggregate 2 Aggregate 1

Aggregate id 2 Aggregate id 1

e2

e5 e6 e3 e7

Aggregate Repository

Aggregate 2 Aggregate 1

Aggregate id 2 Aggregate id 1

e5

e6 e7

Aggregate Repository

Aggregate 2 Aggregate 1

Aggregate id 2 Aggregate id 1

e6

Aggregate Repository

Aggregate 2 Aggregate 1

Aggregate Repository

Command

Aggregate 1

Command Handler

Aggregate Repository

Command Handler

e8 e9

Aggregate Repository

e8 e9 Aggregate 1

Aggregate Repository

Aggregate 1 e8 e9

Aggregate Repository

Aggregate 1 e9

Aggregate Repository

Aggregate 1

Aggregate Repository Event Store

Update

append events where rowId =

e8 e9

Event Store

Aggregate id 1 Aggregate id 2 Row ID Events

e1

e2 e3 e5 e6 e7 e4 e8 e9

Aggregate Repository

Query

Aggregate Service

Get all aggregates for a customer

Aggregate Repository Aggregate Service

Response Aggregate 2 Aggregate 1

Aggregate

Downloaded Aggregate License Aggregate

License Service Acquire License Endpoint

Query

customerId: Bayley titleId: GlowS01E01

License Service

Query

customerId: Bayley titleId: GlowS01E01

Downloaded Service

isAllowed?

Query

customerId: Bayley titleId: GlowS01E01 since: 6/27/2016

Downloaded Service Downloaded Repository

getAggregates

Downloaded Service Downloaded Repository Response Downloaded Aggregate customerId: Bayley titleId: GlowS01E01 date: 2/15/2017 Downloaded Aggregate customerId: Bayley titleId: GlowS01E01 date: 5/25/2017

License Service Downloaded Service Response

isAllowed?

aggregates.size() < yearly limit

License Service Downloaded Service Response

isAllowed?

True

aggregates.size() < yearly limit (2) < (3)

License Service

Update

License Repository

Create Aggregate

Create License Command customerId: Bayley titleId: GlowS01E01 date: 6/27/2017 License Aggregate customerId: titleId: expires: released:

License Repository

Command Handler

Create License Command customerId: Bayley titleId: GlowS01E01 date: 6/27/2017 License Aggregate customerId: titleId: expires: released:

License Repository

Command Handler

Create License Command customerId: Bayley titleId: GlowS01E01 date: 6/27/2017 License Aggregate customerId: titleId: expires: released:

License Created Event

customerId: Bayley titleId: GlowS01E01 date: 6/27/2017

License Repository

Event Handler

License Aggregate customerId: titleId: expires: released:

License Created Event

customerId: Bayley titleId: GlowS01E01 date: 6/27/2017 License Aggregate customerId: Bayley titleId: GlowS01E01 expires: 7/27/2017 released: false

License Repository

Event Store

Save

License Created Event

customerId: Bayley titleId: GlowS01E01 date: 6/27/2017

License Service License Repository

Create Aggregate

Created

Acquire License Endpoint

Response

License Service License Repository Response

isAllowed?

aggregates.filter(GlowS01E01) .size() < yearly limit

Get all licenses

Table

Partition Keys Clustering Columns Columns

Event Table

rowId

text

Event Time

timestamp

Event Data

blob

Aggregate Id

text

Event Mapper

String

• A fast and efficient object graph serialization framework for Java

○ https://github.com/EsotericSoftware/kryo

• Able to preconfigure the library to your data models for extra compaction
• Custom serializers

AggregateId

text

Event Data

blob

rowId

text

Event Time

timestamp

House of Cards LicenseCreated

Matt:LicenseAggregate

Buddy Thunderstruck LicenseCreated 1 The Hardy Boys LicenseCreated

Jeff:LicenseAggregate

2

AggregateId

text

Event Data

blob

rowId

text

Event Time

timestamp

House of Cards LicenseCreated

Matt:LicenseAggregate

House of Cards

LicenseReleased

3 Buddy Thunderstruck LicenseCreated 1 The Hardy Boys LicenseCreated

Jeff:LicenseAggregate

2

AggregateId

text

Event Data

blob

rowId

text

Event Time

timestamp

House of Cards LicenseCreated

Matt:LicenseAggregate

House of Cards

LicenseReleased

3 Buddy Thunderstruck LicenseCreated 1 The Hardy Boys LicenseCreated

Jeff:LicenseAggregate

2

AggregateId

text

Event Data

blob

rowId

text

Event Time

timestamp

House of Cards LicenseCreated

Matt:LicenseAggregate

House of Cards

LicenseReleased

3 Buddy Thunderstruck LicenseCreated 1 The Hardy Boys LicenseCreated

Jeff:LicenseAggregate

2

Snapshotting

House of Cards LicenseCreated Matt:LicenseAggregate House of Cards LicenseReleased 3 Buddy Thunderstruck LicenseCreated 1 Master of None LicenseCreated Master of None LicenseReleased 3 Bojack LicenseCreated 1 Lucha Underground LicenseCreated Lucha Underground LicenseReleased 3 Grace & Frankie LicenseCreated 1 House of Cards LicenseCreated

Snapshot Table

rowId

text

version

int

Snapshot Data

blob

Version

int

Event Data

blob

rowId

text

1 binaryData

Matt:LicenseAggregate

AggregateId

text

Event Data

blob

rowId

text

Event Time

timestamp

House of Cards LicenseCreated

Matt:LicenseAggregate:0

House of Cards

LicenseReleased

Matt:LicenseAggregate:1

3 Jessica Jones LicenseCreated 1 The Hardy Boys LicenseCreated

Jeff:LicenseAggregate:0

2

Working With Event Sourcing

Flexibility in Practice

• Changes to the Data Model are trivial!

○ Device deactivation

New Device Deactivation Requirement

Deactivate Device Endpoint Device Service Device Repository Device Aggregate

Device Deactivated Event

Deactivate Device Command

New Device Deactivation Requirement...

Deactivate Device Endpoint Device Service

Deactivate Device

Can Deactivate?

Device Repository

Response

Device Aggregate

CustomerId: Bayley DeviceId: iPhone Deactivations: 1

New Device Deactivation Requirement...

Deactivate Device Endpoint Device Service

Deactivate Device

Can Deactivate

License Service

Release Licenses

Device Repository

Command Handler

Deactivate Device Command

customerId: Bayley deviceId: iPhone

Device Deactivated Event

customerId: Bayley deviceId: iPhone

Device Aggregate

CustomerId: Bayley DeviceId: iPhone Deactivations: 1

New Device Deactivation Requirement...

Debugging

• Possibly the biggest win
• Current tooling is rudimentary

{ "Event" : "LicenseAcquiredEvent", "Event Time" : "2017-06-25 05:23:00 PM", "aggregateId" : "aggregateId1", "committed" : true, "eventTimeStamp" : 1498411380903, "eventOrderNumber" : 0 }

Debugging...

Acquire License Event

customerId: Bayley deviceId: iPhone date: 01/05/2017

Release License Event

customerId: Bayley deviceId: iPhone date: 01/05/2017

Renew License Event

customerId: Bayley deviceId: iPhone date: 01/06/2017

License Aggregate

customerId:Bayley expires: 01/07/2017 released: True

Reliability: Fallbacks

Edge API Layer License Accounting Service Client

License Accounting Service

Cassandra

Fallback Response

How Did it Scale?

Predictive Scaling

Scalability with Cassandra SSD (I2) Nodes

Scalability with Cassandra SSD Nodes...

• Could handle much higher loads
• Storage use ramping up very quickly

Storage Optimizations

• TTLs
• HDD (D2) clusters:

○ More storage ○ Higher latency (up to 1 second)

Storage Optimizations...

• Partitioned approach

○ Snapshot and subsequent events saved to i2 (SSD) cluster ○ Archive events to D2 (HDD) cluster

Snapshot Data

Version 3

Event Data

Version 3

Event Data

Version <= 2

i2 D2

Storage Optimizations...

Partitioned approach using CQRS (Command Query Responsibility Segregation)

• Write segregation

○ Event handlers that determine partitioned usage

• Query segregation

○ Query SSD or HDD (or both) ○ Uninterrupted event stream

Event Data

i2

D2

Snapshot + Events

Event Handler

Key Event Sourcing Take-Aways

• Flexible: Adapting to change can be simple

Key Event Sourcing Take-Aways

• Flexible: Adapting to change can be simple
• Debugging: Debugging data / state transitions greatly simplified

Key Event Sourcing Take-Aways

• Flexible: Adapting to change can be simple
• Debugging: Debugging data / state transitions greatly simplified
• Reliable: Fallbacks provide service reliability

Key Event Sourcing Take-Aways

• Flexible: Adapting to change can be simple
• Debugging: Debugging data / state transitions greatly simplified
• Reliable: Fallbacks provide service reliability
• Scalable: Service scaled well, but good architectural solutions for

data storage solutions should be considered.

Questions?

Phillipa Avery

Senior Software Engineer Project Technical Lead and Engineer Downloads License Accounting pippa@netflix.com / @PhillipaAvery

Robert Reta

Senior Software Engineer Event Sourcing System Architect Downloads License Accounting rreta@netflix.com / @rreta04