Scaling Dropbox P R E S L AV L E , N O V E M B E R 7 T H , 2 0 1 - - PowerPoint PPT Presentation

▶
scaling dropbox
SMART_READER_LITE
LIVE PREVIEW

Scaling Dropbox P R E S L AV L E , N O V E M B E R 7 T H , 2 0 1 - - PowerPoint PPT Presentation

Feb 17, 2023 18 likes •743 views

Scaling Dropbox P R E S L AV L E , N O V E M B E R 7 T H , 2 0 1 6 Zone Zone (west) (east) Zone (central) block.dropbox.com Zone Zone (west) (east) Zone (central) block.dropbox.com Zone Zone (west) (east) Zone (central)

slide-1
SLIDE 1

Scaling Dropbox

P R E S L AV L E , N O V E M B E R 7 T H , 2 0 1 6

slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4

block.dropbox.com

Zone (west) Zone (east) Zone (central)

slide-5
SLIDE 5

block.dropbox.com

Zone (west) Zone (east) Zone (central)

slide-6
SLIDE 6

block.dropbox.com

Zone (west) Zone (east) Zone (central)

slide-7
SLIDE 7

Fear of the unknown

slide-8
SLIDE 8

M E M O R Y L E A K

slide-9
SLIDE 9

S Y N C H O R N I Z AT I O N E V E N T

slide-10
SLIDE 10

Success story

slide-11
SLIDE 11

TO DAY ’ S TA L K

  • 2012
  • SCALING CHALLENGES
  • 2016
  • Q&A
slide-12
SLIDE 12

P R E S L AV L E

  • At Dropbox since 2013
  • Projects: Magic Pocket,

Infrastructure Performance, Traffic team

slide-13
SLIDE 13

F I L E , S Y N C & S H A R E

slide-14
SLIDE 14

5 0 0 M I L L I O N U S E R S

slide-15
SLIDE 15

2 0 1 2

metaserver metaserver metaserver blockserver blockserver blockserver S3 DB DB DB Memcached Memcached Memcached nginx nginx LB notification server clients nginx nginx LB async processing async processing async processing

AWS Dropbox’s datacenters

slide-16
SLIDE 16

B LO C K DATA I N S 3

metaserver metaserver metaserver blockserver blockserver blockserver S3 DB DB DB Memcached Memcached Memcached nginx nginx LB notification server clients nginx nginx LB async processing async processing async processing

AWS Dropbox’s datacenters AWS

slide-17
SLIDE 17

M E TA DATA I N M Y S Q L

metaserver metaserver metaserver blockserver blockserver blockserver S3 DB DB DB Memcached Memcached Memcached nginx nginx LB notification server clients nginx nginx LB async processing async processing async processing

AWS Dropbox’s datacenters Dropbox’s datacenters

slide-18
SLIDE 18

1 . F E TC H M E TA DATA

metaserver metaserver metaserver blockserver blockserver blockserver S3 DB DB DB Memcached Memcached Memcached nginx nginx LB notification server clients nginx nginx LB async processing async processing async processing

AWS Dropbox’s datacenters

metaserver DB LB clients Memcached

slide-19
SLIDE 19

2 . D OW N LOA D B LO C K S

metaserver metaserver metaserver blockserver blockserver blockserver S3 DB DB DB Memcached Memcached Memcached nginx nginx LB notification server clients nginx nginx LB async processing async processing async processing

AWS Dropbox’s datacenters

blockserver S3 LB LB clients

slide-20
SLIDE 20

3 . WA I T F O R N OT I F I C AT I O N S

metaserver metaserver metaserver blockserver blockserver blockserver S3 DB DB DB Memcached Memcached Memcached nginx nginx LB notification server clients nginx nginx LB async processing async processing async processing

AWS Dropbox’s datacenters

notification server clients metaserver

slide-21
SLIDE 21

P Y T H O N E V E R Y W H E R E

metaserver metaserver metaserver blockserver blockserver blockserver S3 DB DB DB Memcached Memcached Memcached nginx nginx LB notification server clients nginx nginx LB async processing async processing async processing

AWS Dropbox’s datacenters

slide-22
SLIDE 22

Dropbox’s datacenters meta-client meta-client meta-client meta-client meta-client meta-web meta-api meta-api meta-api meta-mobile meta-mobile meta-mobile

C LU S T E R I S O L AT I O N

slide-23
SLIDE 23

Scaling Databases Scaling as Organization Scaling Software Managing Complexity

slide-24
SLIDE 24

S C A L I N G DATA BA S E S

mysql master mysql replica mysql replica metaserver Memcached Memcached Memcached shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

…

slide-25
SLIDE 25

H O R I ZO N TA L S C A L I N G

shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

… …

metaserver metaserver metaserver metaserver metaserver metaserver

slide-26
SLIDE 26

CO N N E C T I O N S

shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

… …

metaserver metaserver metaserver metaserver metaserver metaserver

slide-27
SLIDE 27

S Q L P R OX Y

shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

… …

metaserver metaserver metaserver metaserver metaserver metaserver SQL Proxy SQL Proxy SQL Proxy

slide-28
SLIDE 28

Scaling as Organization Scaling Software Managing Complexity Scaling Databases

slide-29
SLIDE 29

G LO BA L DATA BA S E

slide-30
SLIDE 30

AVA I L A B I L I T Y I S S U E S

slide-31
SLIDE 31

P L AY B O O K

1. Check for ongoing deployments or newly enabled features

slide-32
SLIDE 32

P L AY B O O K

1. Check for ongoing deployments or newly enabled features 2. Check for recently started background jobs

slide-33
SLIDE 33

1. Check for ongoing deployments or newly enabled features 2. Check for recently started background jobs 3. DBA oncall, please help!

P L AY B O O K

slide-34
SLIDE 34

Dropbox grew from 100 to 500 employees

slide-35
SLIDE 35
  • Slow queries would adversely impact performance across the board
slide-36
SLIDE 36
  • Slow queries would adversely impact performance across the board
  • More features => Managing more independent MySQL
slide-37
SLIDE 37
  • Slow queries would adversely impact performance across the board
  • More features => Managing more independent MySQL
  • Reactively (re)sharding individual databases as they hit capacity
slide-38
SLIDE 38
  • Slow queries would adversely impact performance across the board
  • More features => Managing more independent MySQL
  • Reactively (re)sharding individual databases as they hit capacity
  • Impacted developer productivity
slide-39
SLIDE 39

S C A L A B L E M E TA DATA S TO R E D E S I G N E D F O R M U LT I -T E N A N C Y

2013 — Present

slide-40
SLIDE 40

S H A R D I N G A N D C AC H I N G B E H I N D T H E S C E N E S

slide-41
SLIDE 41

E N T I T I E S A N D A S S O C I AT I O N S

slide-42
SLIDE 42

F I R S T G O S E R V I C E

slide-43
SLIDE 43

Scaling Software Scaling as Organization Managing Complexity Scaling Databases

slide-44
SLIDE 44
slide-45
SLIDE 45

P E R F E C T S TO R M

slide-46
SLIDE 46

S H A R D I N G

slide-47
SLIDE 47

P H OTO A L B U M S

slide-48
SLIDE 48

T E A M A D M I N CO N S O L E

slide-49
SLIDE 49

R E Q U E S T FA N O U T

request

slide-50
SLIDE 50

Colocation ID Counter 8 bytes 8 bytes

G LO BA L I D

  • Colocation ID: Identifies a shard
  • Counter: Unique ID within the shard
slide-51
SLIDE 51

Lack of colocation also hurts performance

slide-52
SLIDE 52

N E W S E R V I C E : F I L E J O U R N A L

shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

… …

metaserver metaserver metaserver metaserver File Journal File Journal File Journal

…

metaserver metaserver

slide-53
SLIDE 53

S H A R D FA I LU R E

shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

… …

metaserver metaserver metaserver metaserver File Journal File Journal File Journal

…

metaserver metaserver shard1 master

slide-54
SLIDE 54

S H A R D I N G ( PA R T I I )

slide-55
SLIDE 55

LO N G T I M E O U T S

shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

… …

metaserver metaserver metaserver metaserver File Journal File Journal File Journal

…

metaserver metaserver shard1 master

slide-56
SLIDE 56

R U N O U T O F W O R K E R S

shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

… …

metaserver metaserver metaserver metaserver File Journal File Journal File Journal

…

metaserver metaserver shard1 master File Journal File Journal File Journal

slide-57
SLIDE 57

C A S C A D I N G FA I LU R E

shard1 master shard1 replica shard1 replica shard0 master shard0 replica shard0 replica shardN master shardN replica shrardN replica

… …

metaserver metaserver metaserver metaserver File Journal File Journal File Journal

…

metaserver metaserver shard1 master File Journal File Journal File Journal metaserver metaserver metaserver metaserver metaserver metaserver

slide-58
SLIDE 58

Limit resources dedicated to processing a single shard

S H A R D I S O L AT I O N

slide-59
SLIDE 59

Managing Complexity Scaling as Organization Scaling Software Scaling Databases

slide-60
SLIDE 60

500PB+ user block data 3+ geographic regions 500+ million users

M AG I C P O C K E T B LO C K S TO R AG E S Y S T E M

slide-61
SLIDE 61

Zone (west) Zone (east) Zone (central) put put put get get get

slide-62
SLIDE 62

complicated!

☹

☺

simple complicated!

☹

complicated!

☹

complicated!

☹

complicated!

☹

complicated!

☹

slide-63
SLIDE 63

P Y T H O N , G O & R U S T

slide-64
SLIDE 64

https://blogs.dropbox.com/tech/

slide-65
SLIDE 65

2 0 1 6

meta-client meta-client meta-client meta-client meta-client meta-web meta-api meta-api meta-api meta-mobile meta-mobile meta-mobile File Journal File Journal File Journal Search Search Search Auth Auth Auth service Block Routing Block Routing Block Routing Auth Auth Edgestore Auth Auth Presence &Notications File Journal File Journal Cape

…

blockserver blockserver blockserver Magic Pocket Magic Pocket Magic Pocket Blockservice Riviera Riviera Thumbnail service

slide-66
SLIDE 66

H O W TO P R E V E N T C A S C A D I N G FA I LU R E ?

meta-client meta-client meta-client meta-client meta-client meta-web meta-api meta-api meta-api meta-mobile meta-mobile meta-mobile File Journal File Journal File Journal Search Search Search Auth Auth Auth service Block Routing Block Routing Block Routing Auth Auth Edgestore Auth Auth Presence &Notications File Journal File Journal Cape

…

blockserver blockserver blockserver Magic Pocket Magic Pocket Magic Pocket Blockservice Riviera Riviera Thumbnail service Search

slide-67
SLIDE 67

BA N DA I D : P E R R O U T E I S O L AT I O N

slide-68
SLIDE 68

Q U E U E P R I O R I T I Z AT I O N

slide-69
SLIDE 69

Partition & Isolate (data or services)

slide-70
SLIDE 70

cluster isolation: data model isolation: shard isolation: region isolation: route isolation: Metaserver Edgestore File Journal Magic Pocket Bandaid

slide-71
SLIDE 71

Isolation

slide-72
SLIDE 72