SLIDE 1
BOAZ AVITAL
Tech Lead, Core Storage @bx
Providing Flexible Consistency Levels with Manhattan at Twitter - - PowerPoint PPT Presentation
Providing Flexible Consistency Levels with Manhattan at Twitter - - PowerPoint PPT Presentation
Providing Flexible Consistency Levels with Manhattan at Twitter Boaz Avital @bx BOAZ AVITAL Tech Lead, Core Storage @bx M A N H AT TA N USING MANHATTAN Self service creation of applications and datasets Large multitenant clusters
SLIDE 2
SLIDE 3
M A N H AT TA N
SLIDE 4
# T W I T T E R M A N H AT TA N
USING MANHATTAN
Self service creation of applications and datasets Large multitenant clusters Seamless global replication Automatic observability and alerting
SLIDE 5
ADOPTION
More use cases need more features
# T W I T T E R M A N H AT TA N
Use cases you designed for Use cases you didn't
SLIDE 6
ADOPTION
More use cases need more features
# T W I T T E R M A N H AT TA N
Use cases you designed for Use cases you didn't
{
Pain!
SLIDE 7
# T W I T T E R M A N H AT TA N
DO ONE THING WELL
Too many running services with slightly different code and tooling
EVERYTHING AND THE KITCHEN SINK
Bloated software that’s not that good at anything
ONE THING ALL THINGS
BUILDING NEW FEATURES
SLIDE 8
# T W I T T E R M A N H AT TA N
A R C H I T E C T U R E
SLIDE 9
ARCHITECTURE: DATA MODEL
# T W I T T E R M A N H AT TA N
“@womeng” 437698567 profile, username partitioning key local key value
SLIDE 10
ARCHITECTURE: DATA MODEL
# T W I T T E R M A N H AT TA N
“@womeng” 437698567 profile, username partitioning key local key value “https:/ /pbs.twimg.com…” 437698567 profile, image “In which our Periscope…” 437698567 tweets, 70309… “Boston, @WomenWhoC…” 437698567 tweets, 70260… “@bx” 53205685 profile, username “https:/ /pbs.twimg.com…” 53205685 profile, image “Strong consistency in…” 53205685 tweets, 710573… “Anyone else having issu…” 53205685 tweets, 709182…
SLIDE 11
ARCHITECTURE: PARTITIONING
# T W I T T E R M A N H AT TA N
437698567 53205685 partitioning key shards
SLIDE 12
ARCHITECTURE: PARTITIONING
# T W I T T E R M A N H AT TA N
437698567 53205685 partitioning key shards
SLIDE 13
ARCHITECTURE: PARTITIONING
# T W I T T E R M A N H AT TA N
437698567 53205685 partitioning key shards
SLIDE 14
ARCHITECTURE: MESSAGING
# T W I T T E R M A N H AT TA N
C C C C
SLIDE 15
ARCHITECTURE: CONSISTENCY
# T W I T T E R M A N H AT TA N
Dynamo + Last Write Wins
SLIDE 16
ARCHITECTURE: CONSISTENCY
# T W I T T E R M A N H AT TA N
Dynamo + Last Write Wins
SLIDE 17
ARCHITECTURE: CONSISTENCY
# T W I T T E R M A N H AT TA N
Dynamo + Last Write Wins
SLIDE 18
ARCHITECTURE: CONSISTENCY
# T W I T T E R M A N H AT TA N
Replica Reconciliation
SLIDE 19
BENEFITS OF EVENTUAL CONSISTENCY
# T W I T T E R M A N H AT TA N
AVAILABILITY LATENCY SIMPLICITY
SLIDE 20
WHEN IT’S NOT ENOUGH
# T W I T T E R M A N H AT TA N
@justinbieber reserve() reserve()
? ?
SLIDE 21
WHEN IT’S NOT ENOUGH
# T W I T T E R M A N H AT TA N
reserve() reserve()
SLIDE 22
WHEN IT’S NOT ENOUGH
# T W I T T E R M A N H AT TA N
@username claimed by 53205685 @username is free @username is free
SLIDE 23
ADAPTING ARCHITECTURE
# T W I T T E R M A N H AT TA N
Logs!
APACHE BOOKKEEPER TWITTER DISTRIBUTEDLOG
SLIDE 24
ADAPTING ARCHITECTURE
# T W I T T E R M A N H AT TA N
Operations
read write compare-and-set increment
SLIDE 25
ADAPTING ARCHITECTURE
# T W I T T E R M A N H AT TA N
Check: @username → free Set: @username → 53205685 Check: @username → free Set: @username → 20719205
SLIDE 26
ADAPTING ARCHITECTURE
# T W I T T E R M A N H AT TA N
C C C C
SLIDE 27
ADAPTING ARCHITECTURE
# T W I T T E R M A N H AT TA N
C C C C
SLIDE 28
ADAPTING ARCHITECTURE
# T W I T T E R M A N H AT TA N
C C C C
SLIDE 29
ADAPTING ARCHITECTURE
# T W I T T E R M A N H AT TA N
C C C C
SLIDE 30
ADAPTING ARCHITECTURE
# T W I T T E R M A N H AT TA N
C C C C
SLIDE 31
RESULTS
# T W I T T E R M A N H AT TA N
WHAT WE’VE GAINED WHAT WE’VE LOST
In order updates for keys (but not full transactions) Failure isolation at shard level Mixing strong and eventual consistency in a cluster Mixing strong and eventual consistency in a dataset A few 10s of milliseconds added to average latency Latency hiccups during failures Potential for stream halts on error
SLIDE 32
# T W I T T E R M A N H AT TA N
C O N S E Q U E N C E S
SLIDE 33
A DIFFERENT CONSISTENCY MODEL
# T W I T T E R M A N H AT TA N
y, v2 y, v2 y, v2 x, v1 x, v1 y, v2 EVENTUAL CONSISTENCY x, v1 y, v2
SLIDE 34
A DIFFERENT CONSISTENCY MODEL
# T W I T T E R M A N H AT TA N
STRONG CONSISTENCY y y y x, a1 x, b1 y, b2 x, c1 y, c2
SLIDE 35
A DIFFERENT CONSISTENCY MODEL
# T W I T T E R M A N H AT TA N
NO MORE REPLICA RECONCILIATION
SLIDE 36
A DIFFERENT CONSISTENCY MODEL
# T W I T T E R M A N H AT TA N
BACKFILL AND DATA MIGRATION Live requests Backfill data
SLIDE 37
A DIFFERENT TOPOLOGY TRANSITION
# T W I T T E R M A N H AT TA N
Old New
SLIDE 38
A DIFFERENT TOPOLOGY TRANSITION
# T W I T T E R M A N H AT TA N
2 1 3
SLIDE 39
A DIFFERENT DEFINITION OF TIME
# T W I T T E R M A N H AT TA N
x, ttl=1001 now=1000 x, ttl=1001 now=1003 x, ttl=1001 now=1002 x ⍉ ⍉
SLIDE 40
A DIFFERENT DEFINITION OF TIME
# T W I T T E R M A N H AT TA N
external time
write value x expire value x check for x, set y write value x expire value x check for x, set y write value x expire value x check for x, set y
SLIDE 41
A DIFFERENT DEFINITION OF TIME
# T W I T T E R M A N H AT TA N
now = 1002 x, ttl=1001 now=1000 x, ttl=1001 now=1002 x, ttl=1001 now=1002
SLIDE 42
WHAT THE USER SEES
# T W I T T E R M A N H AT TA N
Provisioning Querying
SLIDE 43
T H A N K YO U
@ B X
SLIDE 44