NoSQL like There is No Tomorrow Khawaja Head of Engineering, NoSQL - - PowerPoint PPT Presentation

@ksshams @swami_79

NoSQL like There is No Tomorrow

Swaminathan Sivasubramanian

GM, NoSQL

Khawaja

Head of Engineering, NoSQL

Swami

@ksshams @swami_79

how can you build your own DynamoDB Scale service?

@ksshams @swami_79

let’s start with a story about a little company called amazon.com

@ksshams @swami_79

• nce upon a time...

(in 2000)

episode 1

@ksshams @swami_79

a few thousand miles away... (seattle)

@ksshams @swami_79

amazon.com - a rapidly growing Internet based retail business relied on relational databases

@ksshams @swami_79

we had 1000s of independent services

@ksshams @swami_79

each service managed its state in RDBMs

@ksshams @swami_79

RDBMs are actually kind of cool

@ksshams @swami_79

first of all... SQL!!

@ksshams @swami_79

so it is easier to query..

@ksshams @swami_79

easier to learn

@ksshams @swami_79

as versatile as a swiss army knife

complex queries

key-value access transactions analytics

@ksshams @swami_79

RDBMs are too similar to Swiss Army Knives

@ksshams @swami_79

but sometimes.. swiss army knifes.. can be more than what you bargained for

@ksshams @swami_79

partitioning

easy

re-partitioning

hard..

@ksshams @swami_79

so we bought

bigger boxes...

@ksshams @swami_79

Q4 was hard-work at Amazon

benchmark new hardware migrate to new hardware repartition databases pray ...

@ksshams @swami_79

RDBMs availability challenges..

@ksshams @swami_79

then.. (in 2005)

episode 2

@ksshams @swami_79

amazon dynamo

predecessor to dynamoDB

specialist tool :

• limited querying capabilities
• simpler consistency

replicated DHT with consistent hashing

• ptimistic replication

“sloppy quorum” anti-entropy mechanism

• bject versioning

@ksshams @swami_79

dynamo had many benefits

• higher availability
• we traded it off for eventual consistency

• incremental scalability
• no more repartitioning
• no need to architect apps for peak
• just add boxes

• simpler querying model ==>> predictable performance

@ksshams @swami_79

but dynamo was not perfect...

lacked strong consistency

@ksshams @swami_79

but dynamo was not perfect...

scaling was easier, but...

@ksshams @swami_79

but dynamo was not perfect...

steep learning curve

@ksshams @swami_79

but dynamo was not perfect...

dynamo was a product ... ==>> not a service...

@ksshams @swami_79

then.. (in 2012)

episode 3

@ksshams @swami_79

“Even though we have years of experience with large, complex NoSQL architectures, we are happy to be finally out of the business of managing it ourselves.” - Don MacAskill, CEO

• NoSQL database
• fast & predictable performance
• seamless scalability
• easy administration

DynamoDB

@ksshams @swami_79

build services not software!!

@ksshams @swami_79

amazon.com’s experience with services

@ksshams @swami_79

how do you create a successful service?

@ksshams @swami_79

with great services, comes great responsibility

@ksshams @swami_79

DynamoDB Goals and Philosophies

never compromise on durability

scale is our problem easy to use

scale in rps consistent and low latencies

@ksshams @swami_79

Architect Test Monitor Goals D e p l

• y

Develop

Customer

@ksshams @swami_79

Architect

Test

Monitor Goals

Customer

Deploy Develop

@ksshams @swami_79

Sacred Tenets in Services

plan for success - plan for scalability don’t compromise durability for performance

plan for failures - fault -tolerance is key

consistent performance is important design - think of blast radius insist on correctness

@ksshams @swami_79

fault tolerance is a lesson best learned offline

@ksshams @swami_79

a simple 2-way replication system of a traditional database…

Primary Standby

Writes

@ksshams @swami_79

@ksshams @swami_79

P S

S ¡is ¡dead, ¡need ¡ to ¡trigger ¡new ¡ replica P ¡is ¡dead, ¡need ¡to ¡ promote ¡myself

P’

@ksshams @swami_79

@ksshams @swami_79

improved Replication: quorum

Replica Replica

Writes

Replica

Quorum: Successful write on a majority

Not so easy..

Replica B Replica C

Writes from client A

Replica A Replica D

New member in the group Should I continue to serve reads? Should I start a new quorum?

Replica E Replica F

Reads and Writes from client B Classic Split Brain Issue in Replicated systems leading to lost writes!

@ksshams @swami_79

Building correct distributed systems is not straight forward..

• How do you handle replica failures?
• How do you ensure there is not a parallel

quorum?

• How do you handle partial failures of replicas?
• How do you handle concurrent failures?

correctness is hard, but necessary

Formal Methods

Formal Methods

to minimize bugs, we must have a precise description of the design

Formal Methods

code is too detailed how would you express partial failures or concurrency? design documents and diagrams are vague & imprecise

Formal Methods

law of large numbers is your friend, so design for scale until you hit large numbers

@ksshams @swami_79

TLA+ to the rescue?

@ksshams @swami_79

PlusCal

@ksshams @swami_79

formal methods are necessary

but not sufficient..

@ksshams @swami_79

A r c h i t e c t

Test

M

• n

i t

• r

Goals

customer

Deploy D e v e l

• p

@ksshams @swami_79

forget to test - no, serious .. don’t ly

embrace failure and don’t be surprised

simulate failures at unit test level

fault injection testing

datacenter testing

network brown out testing

scale testing

testing is a lifelong journey

@ksshams @swami_79

testing is necessary but not sufficient..

@ksshams @swami_79

Architect Test Monitor G

• a

l s

Customer

Deploy D e v e l

• p

@ksshams @swami_79

release cycle

gamma simulate real world

• ne box

does it work? phased deployment treading lightly monitor does it still work?

@ksshams @swami_79

Canaries

@ksshams @swami_79

Alarms

@ksshams @swami_79

Monitor customer behavior

Architect T e s t

Monitor

G

• a

l s

Customer

Deploy Develop

@ksshams @swami_79

measuring customer experience is key

don’t be satisfied by average - look at 99 percentile

@ksshams @swami_79

understand the scaling dimensions

@ksshams @swami_79

understand how your service will be abused

@ksshams @swami_79

let’s see these rules in action through a true story

@ksshams @swami_79

we were building distributed systems all over amazon.com

@ksshams @swami_79

we needed a uniform and correct way to do consensus..

@ksshams @swami_79

so we built a paxos lock library service

@ksshams @swami_79

such a service is so much more useful than just leader election.. it became a distributed state store

@ksshams @swami_79

such a service is so much more useful than just leader election..

• r a distributed state store

wait wait.. you’re telling me if I poll, I can detect node failure?

@ksshams @swami_79

we acted quickly - and scaled up our entire fleet with more nodes

doh!!!!

we slowed consensus...

@ksshams @swami_79

understand the scaling dimensions

& scale them independently...

@ksshams @swami_79

L e a d e r E l e c t i

• n

Failure Notification

State Store

a lock service has 3 components..

@ksshams @swami_79

L e a d e r E l e c t i

• n

F a i l u r e N

• t

i f i c a t i

• n

State Store

they must be scaled independently..

@ksshams @swami_79

L e a d e r E l e c t i

• n

F a i l u r e N

• t

i f i c a t i

• n

State Store

they must be scaled independently..

@ksshams @swami_79

L e a d e r E l e c t i

• n

F a i l u r e N

• t

i f i c a t i

• n

State Store

they must be scaled independently..

@ksshams @swami_79

understand scaling dimensions

• bserve

how service is used

• scalability over features

strive for correctness relentlessly test monitor like a hawk

@ksshams @swami_79

Thank You!

@swami_79 @kshams