Solving Everyday Data Problems with FoundationDB Ryan Worl - - PowerPoint PPT Presentation

▶
solving everyday data problems with foundationdb
SMART_READER_LITE
LIVE PREVIEW

Solving Everyday Data Problems with FoundationDB Ryan Worl - - PowerPoint PPT Presentation

Nov 18, 2022 40 likes •562 views

Solving Everyday Data Problems with FoundationDB Ryan Worl (ryantworl@gmail.com) Consultant About Me Independent software engineer Todays real example is from ClickFunnels > 70,000 customers, > 1.8B of payments processed

slide-1
SLIDE 1

Solving Everyday Data Problems with FoundationDB

Ryan Worl (ryantworl@gmail.com) Consultant

slide-2
SLIDE 2

About Me

  • Ryan Worl
  • @ryanworl on Twitter
  • ryantworl@gmail.com
  • Independent software engineer
  • Today’s real example is from ClickFunnels
  • > 70,000 customers, > 1.8B of payments processed
  • Billions of rows of OLTP data (Amazon Aurora MySQL)
slide-3
SLIDE 3

Agenda

  • How FoundationDB Works
  • “Everyday” data problems
  • Why FoundationDB can be the solution
  • ClickFunnels’ recent data problem
  • FoundationDB for YOUR data problems
slide-4
SLIDE 4

1 Coordinators elect & heartbeat Cluster Controller (Paxos)

Coordinators Cluster Controller

Coordinators store core cluster state, used like ZooKeeper All processes register themselves with the Cluster Controller

slide-5
SLIDE 5

2 Cluster Controller (CC) assigns Master Role

Master

slide-6
SLIDE 6

3 CC assigns TLog, Proxy, Resolver, and Storage Roles

Proxy

slide-7
SLIDE 7

3 CC assigns TLog, Proxy, Resolver, and Storage Roles

Resolver

slide-8
SLIDE 8

3 CC assigns TLog, Proxy, Resolver, and Storage Roles

TLog

slide-9
SLIDE 9

3 CC assigns TLog, Proxy, Resolver, and Storage Roles

Storage

slide-10
SLIDE 10

4 On Start: Your App Connects and Asks CC For Topology

YOUR APP

slide-11
SLIDE 11

4 Client Library Asks a Proxy for Key Range to Storage Mapping

YOUR APP

slide-12
SLIDE 12

4 Data Distribution Runs On Master, Key Map Stored in Database

FF

YOUR APP

slide-13
SLIDE 13

5 Start a Transaction: Ask Master for Latest Version

YOUR APP

slide-14
SLIDE 14

5 Start a Transaction: Ask Master for Latest Version (Batched)

YOUR APP 100

slide-15
SLIDE 15

6 Perform Reads at Read Version Directly to Storage

YOUR APP

00 55 AF FF

slide-16
SLIDE 16

Consequences

  • All replicas participate in reads
  • Client load balances among different replicas
  • Failures of all but one replica for each range keep the

system alive

@ryanworl

slide-17
SLIDE 17

7 Buffer Writes Locally Until Commit

00 55 AF FF

YOUR APP

00 55 AF FF

slide-18
SLIDE 18

8 Commit Part 1: Send R/W Conflict Ranges + Mutations to Proxy

YOUR APP

00 55 AF FF 00 55 AF FF

slide-19
SLIDE 19

8 Part 2: Proxy Batches Txns to Master To Get Commit Version

A H T Z YOUR APP

00 55 AF FF 00 55 AF FF

slide-20
SLIDE 20

Consequences

  • Master is not a throughput bottleneck
  • Intelligent batching makes Master workload small
  • Conflict ranges and mutations are not sent to Master at all

@ryanworl

slide-21
SLIDE 21

8 Part 3: Send Conflict Ranges to Resolvers for Conflict Detection

A H T Z YOUR APP

00 55 AF FF 00 55 AF FF

slide-22
SLIDE 22

8 Part 4: If Isolation Passes, Send Mutations to Relevant TLogs

A H T Z YOUR APP

00 55 AF FF 00 55 AF FF

slide-23
SLIDE 23

8 Part 5: (Async) Storages Pull Mutations from Their Buddy TLogs

A H T Z YOUR APP

00 55 AF FF 00 55 AF FF

slide-24
SLIDE 24

9 Failure Detection: Cluster Controller Heartbeats

A H T Z YOUR APP

00 55 AF FF 00 55 AF FF

slide-25
SLIDE 25

10

Initiate Recovery on Any Transaction Role Failure

A H T Z YOUR APP

฀฀

00 55 AF FF 00 55 AF FF

slide-26
SLIDE 26

฀฀

11

Cluster Controller Failure: Coordinators Elect New One

A H T Z YOUR APP

00 55 AF FF 00 55 AF FF

slide-27
SLIDE 27

12

Storage Server Failure: No Recovery, Repair in Background

A H T Z YOUR APP

฀฀

00 55 AF FF 00 55 AF FF

slide-28
SLIDE 28

Status Quo

  • Most apps start uncomplicated
  • One database, one queue
  • … five years later, a dozen data systems

@ryanworl

slide-29
SLIDE 29

“Everyday” Data Problems?

@ryanworl

https://twitter.com/coda

slide-30
SLIDE 30

“Microservices”

@ryanworl

  • Can make this worse

Service A Service B Service C

slide-31
SLIDE 31

Why is this a problem?

  • Operational costs

○ Administrative costs ○ Duplicated data

  • Development costs

○ Atomicity mostly ignored in the real world ○ Corrupted data extremely common

@ryanworl

slide-32
SLIDE 32

Why is this a problem?

  • Security costs

○ More systems = More risk

  • Error handling never exercised

○ “De-coupled”, “redundant”, “fault tolerant” services mostly a myth

@ryanworl

slide-33
SLIDE 33

Why is this a problem?

  • “Managed cloud services”

○ They will never pick up the pieces ○ They will reboot the machine for you… ○ A weak system run by someone else is still weak ■ e.g. data loss from async replication

@ryanworl

slide-34
SLIDE 34

Why is FoundationDB a solution?

  • Build anything you want or need
  • Multiple systems in one cluster
  • Eventually consistent models easier to build too

○ OLTP Change Log OLAP

@ryanworl

slide-35
SLIDE 35

ClickFunnels’ Recent Data Problem

slide-36
SLIDE 36

The Everyday Data Problem

  • “Smart Lists” based on user-defined rules
  • Running against billions of rows in an OLTP database
  • Both user-facing and automated (100s of QPS)
slide-37
SLIDE 37

The Everyday Data Problem

SELECT contacts.id from contacts LEFT JOIN emails on emails.contact_id = contacts.id LEFT JOIN templates on templates.id = emails.template_id WHERE ...

slide-38
SLIDE 38

Breaking it down

  • Data volume = 100s of GB
  • Complex joins and row-oriented storage
  • Indexes can’t satisfy every query efficiently
  • Aurora = single threaded queries
  • Really just set operations on integers at the core...

@ryanworl

slide-39
SLIDE 39

Bitmap indexes!

slide-40
SLIDE 40

Bitmap Indexes 101

  • Roaring Bitmap Library (roaringbitmap.org)
  • Space usage proportional to number of set bits
  • Billions of operations per second (SIMD)
  • Easily parallelizable (multi-core and distributed)

@ryanworl

slide-41
SLIDE 41

Bitmap Indexes 101

  • Multi-minute evaluation times of rules in Aurora
  • Under 100ms with bitmaps

@ryanworl

slide-42
SLIDE 42

New Possibilities

  • Evaluating rules on every customer website page view
  • “How many people will this rule match?” in real time
  • Stats and analytics can adapt to this format
  • E.g. unique email opens per hour with rules applied for

fancy charts

  • Pages load instantly even for the largest customers

@ryanworl

slide-43
SLIDE 43

How to get there - Step One

  • Replicate Aurora binlog into FoundationDB
  • Write volume not high enough to worry about sharding
  • Example of a log structure

[“binlog”, VersionStamp] => MySQL Binlog as JSON

@ryanworl

slide-44
SLIDE 44

How to get there - Step Two

  • Chunk bitmaps into small segments (2^18 is fine)
  • Evaluate rules, set bits where rules match
  • One writer at a time for low contention
  • Example of storing a large object among many keys

[“bitmaps”, rule_id, chunk_id] => Bitmap Chunk

slide-45
SLIDE 45

How to get there - Step Three

  • Do a range read for every chunk for each rule
  • Parallelize by evaluating different ranges
  • Classic fork-join pattern

[“bitmaps”, rule_1, chunk_1] => Chunk [“bitmaps”, rule_2, chunk_1] => Chunk [“bitmaps”, rule_1, chunk_N] => Chunk [“bitmaps”, rule_2, chunk_N] => Chunk

CORE 1 CORE 2

slide-46
SLIDE 46

Experimental Results

  • Real-world queries take 100ms
  • One large box today
  • Distributed later with little extra work
  • HA from auto-scaling group and load balancer
  • < 3000 lines of JavaScript + RoaringBitmap

@ryanworl

slide-47
SLIDE 47

YOUR Everyday Data Problems

  • FoundationDB’s performance

○ Concurrency Potential ○ Coordination Avoidance

  • Break down the transaction critical path

@ryanworl

slide-48
SLIDE 48

https://en.wikipedia.org/wiki/Amdahl%27s_law

slide-49
SLIDE 49

~ 275 allocations / second 160ms latency and growing

https://www.activesphere.com/blog/2018/08/05/high-contention-allocator

slide-50
SLIDE 50

> 3500 allocations / second ~ 13ms latency @ high concurrency

https://www.activesphere.com/blog/2018/08/05/high-contention-allocator

slide-51
SLIDE 51

YOUR Everyday Data Problems

  • Tables, logs, queues, secondary indexes
  • Simple to implement with little code
  • Freedom to build your exact solution
  • … without the explosion of data systems
  • One cluster to manage

@ryanworl

slide-52
SLIDE 52

Questions

  • Email or tweet me if you have questions or want to talk

about specific use cases for FoundationDB

@ryanworl ryantworl@gmail.com