PAT HELLAND AND ME HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS - - PowerPoint PPT Presentation

▶

pat helland and me

PAT HELLAND AND ME HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS - - PowerPoint PPT Presentation

Sep 04, 2022 369 likes •882 views

PAT HELLAND AND ME HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS THAT CAN SCALE ALMOST INFINITELY PAT HELLAND AND ME SEAN T. ALLEN VP OF ENGINEERING AT WALLAROO LABS MEMBER OF THE PONY CORE TEAM AUTHOR OF STORM APPLIED LOVER OF FRENCH

slide-1

SLIDE 1

PAT HELLAND AND ME

HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS THAT CAN SCALE ALMOST INFINITELY

slide-2

SLIDE 2

PAT HELLAND

AND ME

slide-3

SLIDE 3

SEAN T. ALLEN

VP OF ENGINEERING AT WALLAROO LABS MEMBER OF THE PONY CORE TEAM AUTHOR OF “STORM APPLIED” LOVER OF FRENCH STREET ART @SEANTALLEN @WALLAROOLABS @PONYLANG

slide-4

SLIDE 4

DATABASES

APPARENTLY, I LIKE TO STICK THEM IN THINGS…

slide-5

SLIDE 5

SOME AXIOMS…

slide-6

SLIDE 6

TO SCALE INFINITELY, WE HAVE TO SCALE HORIZONTALLY

slide-7

SLIDE 7

TO SCALE INFINITELY, WE MUST AVOID COORDINATION

slide-8

SLIDE 8

DISTRIBUTED TRANSACTIONS ARE A FORM OF COORDINATION

slide-9

SLIDE 9

THEREFORE… TO SCALE INFINITELY, WE CAN’T USE TRANSACTIONS

slide-10

SLIDE 10

WELCOME TO DISTRIBUTED SYSTEMS!

O, BY THE WAY, ALL THE RULES HAVE CHANGED

slide-11

SLIDE 11

WHAT IS SCALING?

slide-12

SLIDE 12

MORE AND MORE THINGS

BUT, THEY DON’T GET

BIGGER. THERE’S JUST…

MORE OF THEM. LOTS MORE.

slide-13

SLIDE 13

WE SCALE ENTITIES

ENTITIES: LIVE ON A SINGLE MACHINE AND ARE MANIPULATED INDIVIDUALLY

slide-14

SLIDE 14

WHAT IS AN ENTITY?

slide-15

SLIDE 15

ENTITIES ARE BOUNDARIES OF ATOMICITY

slide-16

SLIDE 16

Bob

6 3 5 8

Alice

4 2 7 1

slide-17

SLIDE 17

Bob

6 3 5 8

Alice

4 2 7 1

slide-18

SLIDE 18

Bob

6 3 5 8

Alice

4 7 1 2

slide-19

SLIDE 19

Bob

6 3 5 8

Alice

4 2 7 1

slide-20

SLIDE 20

Bob

6 3 5 8

Alice

4 2 7 1

slide-21

SLIDE 21

DENORMALIZE..

ALL THE THINGS!

slide-22

SLIDE 22

TWO LAYER ARCHITECTURE

slide-23

SLIDE 23

scale-agnostic scale-aware API

slide-24

SLIDE 24

scale-agnostic scale-aware API

slide-25

SLIDE 25

scale-agnostic scale-aware API

slide-26

SLIDE 26

scale-agnostic scale-aware API

slide-27

SLIDE 27

scale-agnostic scale-aware API

slide-28

SLIDE 28

scale-agnostic scale-aware API

slide-29

SLIDE 29

TO SCALE INFINITELY, YOUR BUSINESS LOGIC HAS TO BE INDEPENDENT OF SCALE

slide-30

SLIDE 30

WALLAROO

SCALE INDEPENDENT COMPUTING FOR PYTHON

slide-31

SLIDE 31

AND IT’S NOT A DATABASE

slide-32

SLIDE 32

ENTITIES

BUT WE CALL THEM… “STATE OBJECTS”

slide-33

SLIDE 33

TWO LAYER ARCHITECTURE

BUT WE CALL IT… “SCALE INDEPENDENCE”

slide-34

SLIDE 34

user supplied logic Wallaroo runtime Wallaroo API

slide-35

SLIDE 35

user supplied logic Wallaroo runtime Wallaroo API

slide-36

SLIDE 36

user supplied logic Wallaroo runtime Wallaroo API

slide-37

SLIDE 37

user supplied logic Wallaroo runtime Wallaroo API

slide-38

SLIDE 38

WHAT’S HARD?

ALL OF IT? YOU’RE BUILDING A DISTRIBUTED SYSTEMS FRAMEWORK

slide-39

SLIDE 39

CAP THEOREM

CONSISTENCY VS AVAILABILITY… YOU CAN’T ESCAPE IT.

slide-40

SLIDE 40

MESSAGE DELIVERY

AT-MOST-ONCE? AT-LEAST- ONCE? EFFECTIVELY-ONCE? EXACTLY-ONCE?

slide-41

SLIDE 41

MESSAGE ORDERING

WILL YOU MAINTAIN THE ORDERING AS YOU RECEIVED IT?

slide-42

SLIDE 42

LOCAL KNOWLEDGE

YOU HAVE TO WORK HARD TO AVOID COORDINATION.

slide-43

SLIDE 43

PROGRAMMING MODEL

YOU CAN GET WITH THIS, OR YOU CAN GET WITH THAT.

slide-44

SLIDE 44

PERFORMANCE

IT’S A WORD IN THE DICTIONARY

slide-45

SLIDE 45

NETWORK OVERHEAD

YOU AREN’T IN LOCAL MEMORY ANYMORE

slide-46

SLIDE 46

DATA SERIALIZATION

LE SIGH…

slide-47

SLIDE 47

VERIFICATION

LET’S NOT GO THERE… THAT’S AN ENTIRE LECTURE SERIES.

slide-48

SLIDE 48

BTW… YOUR MULTI-CORE COMPUTER

IS ALSO A DISTRIBUTED SYSTEM. BUT THAT’S A STORY FOR ANOTHER DAY.

slide-49

SLIDE 49

LEARN MORE

GITHUB.COM/SEANTALLEN/ PAT-HELLAND-AND-ME