again? Tanya Reilly (@whereistanya) MY SKILLZ Where are you in - - PowerPoint PPT Presentation

▶
again
SMART_READER_LITE
LIVE PREVIEW

again? Tanya Reilly (@whereistanya) MY SKILLZ Where are you in - - PowerPoint PPT Presentation

Feb 25, 2023 274 likes •869 views

Have you tried turning it off and turning it on again? Tanya Reilly (@whereistanya) MY SKILLZ Where are you in RESUME the stack? CMS MY SKILLZ Where are you in RESUME the stack? CMS MY SKILLZ Where are you in RESUME the stack? CMS

slide-1
SLIDE 1

Have you tried turning it off and turning it on again?

Tanya Reilly (@whereistanya)

slide-2
SLIDE 2

Where are you in the stack?

CMS

MY SKILLZ RESUME

slide-3
SLIDE 3

Where are you in the stack? MY SKILLZ RESUME

CMS

slide-4
SLIDE 4

Where are you in the stack? MY SKILLZ RESUME

CMS CMS ???

slide-5
SLIDE 5

Where are you in the stack?

CMS!!

CMS ???

MY SKILLZ RESUME

slide-6
SLIDE 6

What is it?

slide-7
SLIDE 7

Abstractions let us specialise

slide-8
SLIDE 8

A slidedoc is a visual document, developed in presentation software, that is intended to be read and referenced instead of projected.

8

slide-9
SLIDE 9
  • fallback plans
  • dependencies
  • what can we do?
slide-10
SLIDE 10

Fallback plans

slide-11
SLIDE 11

11

backups

Of course! I copied it to /tmp/ database.bak

Backup_Backup_Backup_-_And_Test_Restores, T-Town Photo Booth CC BY 2.0

Backup, Backup Backup and Test Restores, T-Town Photo Booth CC BY 2.0

slide-12
SLIDE 12

If you haven't tested your backups, you don't have backups.

slide-13
SLIDE 13

the identical(ish!) failover site

Take two, they’re small

slide-14
SLIDE 14
  • 1. Is it real?
slide-15
SLIDE 15
  • 2. Is it up to date?

Retro, Melinda Seckington CC BY 2.0

slide-16
SLIDE 16
  • 3. Is the idea of actually using it kind of terrifying?

FreeImages.com/barun patro

slide-17
SLIDE 17

Two sites? I’ve forgotten how to count that low

replicated everything

slide-18
SLIDE 18
  • fallback plans
  • dependencies
  • what can we do?
slide-19
SLIDE 19

(Micro)services

slide-20
SLIDE 20

(Micro)services

slide-21
SLIDE 21

Jenga Tower of 50 blocks, Johannes Bader CC BY 2.0

slide-22
SLIDE 22

Everybody's backend is someone else's frontend

slide-23
SLIDE 23

“

"A service cannot be more available than the intersection of all its critical dependencies."

  • - "The Calculus of Service Availability"

Ben Treynor, Mike Dahlin, Vivek Rau, Betsy Beyer

slide-24
SLIDE 24

Your stack… is really more of a "pile", isn't it?

dependency cycles

slide-25
SLIDE 25

uh...

control plane cycles

ŠJů CC BY 4.0

slide-26
SLIDE 26

26

Why would we ever restart it?

machines that run forever (until they don't)

slide-27
SLIDE 27

27

root@X1:/# halt Connection to x1 closed by remote host. root@x3:/# halt Connection to x3 closed by remote host. $ root@x2:/# halt Connection to x2 closed by remote host. root@x10000:/# halt Connection to x10000 closed by remote host $

Global simultaneous reboot (doesn’t usually look like this)

root@x10001:/# halt Connection to x10001 closed by remote host. $

slide-28
SLIDE 28

tick tick tick...

slide-29
SLIDE 29

tick tick tick...

slide-30
SLIDE 30

tick tick tick...

slide-31
SLIDE 31

tick tick tick...

slide-32
SLIDE 32

tick tick tick...

slide-33
SLIDE 33

tick tick tick...

slide-34
SLIDE 34

tick tick tick...

slide-35
SLIDE 35

tick tick tick...

slide-36
SLIDE 36

tick tick tick...

slide-37
SLIDE 37

tick tick tick...

slide-38
SLIDE 38
  • fallback plans
  • dependencies
  • what can we do?
slide-39
SLIDE 39

dependency discovery

We're built

  • n top of...

really? Are you sure?

It’s Turtles All The Way Down, William Warby. CC BY 2.0

slide-40
SLIDE 40

Original design. "We keep an in-memory index of the available cats with some metadata about them. The cat chooser uses the information from the user's request to determine the attributes that needs to be fulfilled. It checks the index for cats that match those attributes. It pulls those cats from storage and returns one to the user. Indexers run at startup and at intervals to update the metadata and make sure we have accurate cat sentiment analysis and geolocation data."

design docs obscure information

slide-41
SLIDE 41

Original design. "We keep an in-memory index of the available cats with some metadata about them. The cat chooser uses the information from the user's request to determine the attributes that needs to be fulfilled. It checks the index for cats that match those attributes. It pulls those cats from storage and returns one to the user. Indexers run at startup and at intervals to update the metadata and make sure we have accurate cat sentiment analysis and geolocation data." User story: cat must be a cat. "Since catness analysis is an expensive operation, we only run the catness analyzer on pictures that have a high likelihood of being served, i.e., those aleady indexed with geolocation and sentiment data. We'll take the index from the cat chooser and run catness analysis over the cats from that queue."

design docs obscure information

slide-42
SLIDE 42

webserver -> cat_chooser cat_chooser -> cat_geotagger cat_chooser -> cat_sentiment_analyzer cat_chooser -> catness_analyzer cat_chooser -> storage catness_analyzer -> cat_chooser catness_analyzer -> storage cat_geotagger -> storage cat_sentiment_analyzer -> storage

lists are better

slide-43
SLIDE 43

digraph { webserver -> cat_chooser cat_chooser -> cat_geotagger cat_chooser -> cat_sentiment_analyzer cat_chooser -> catness_analyzer cat_chooser -> storage catness_analyzer -> cat_chooser catness_analyzer -> storage cat_geotagger -> storage cat_sentiment_analyzer -> storage }

pictures are even better

slide-44
SLIDE 44

digraph { webserver -> cat_chooser cat_chooser -> cat_geotagger cat_chooser -> cat_sentiment_analyzer cat_chooser -> catness_analyzer cat_chooser -> storage catness_analyzer -> cat_chooser catness_analyzer -> storage cat_geotagger -> storage cat_sentiment_analyzer -> storage }

pictures are even better

$ /usr/bin/dot -T png -o cats.png cats.dot

slide-45
SLIDE 45

But don't put humans in charge of remembering what connects to what

slide-46
SLIDE 46

dependency policies

This is Site Reliability. THERE ARE RULES.

slide-47
SLIDE 47

Are you on the guest list?

policy enforcement

slide-48
SLIDE 48

policy enforcement

Are you on the guest list?

kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: access-nginx spec: podSelector: matchLabels: run: nginx ingress:

  • from:
  • podSelector:

matchLabels: access: "true"

slide-49
SLIDE 49

Manage your dependencies early (It’s harder later on)

Some sock yarn, Kara, CC BY-ND 2.0

slide-50
SLIDE 50

architecting in layers

Look, an actual stack!

slide-51
SLIDE 51

soft dependencies

slide-52
SLIDE 52

The appearance of U.S. Department of Defense (DoD) visual information does not imply or constitute DoD endorsement.

The reserve parachute is always packed by an expert

reliable fallback plans

slide-53
SLIDE 53

53

testing

Really, try turning it

  • ff and

turning it on again

slide-54
SLIDE 54

in conclusion...

slide-55
SLIDE 55

❏ small fallback plans ❏ test everything ❏ manage dependencies ❏ architect in layers

fallback plans: small, simple, solid

slide-56
SLIDE 56

if it's not tested, assume it doesn't work

❏ small fallback plans ❏ test everything ❏ manage dependencies ❏ architect in layers

if it's not tested, assume it doesn't work

slide-57
SLIDE 57

manage your dependencies before you need to

❏ small fallback plans ❏ test everything ❏ manage dependencies ❏ architect in layers

slide-58
SLIDE 58

❏ small fallback plans ❏ test everything ❏ manage dependencies ❏ architect in layers

dependencies go down the stack

slide-59
SLIDE 59

hope is not a strategy

@whereistanya

https://github.com/whereistanya/graphviz/

Presentation template by slidescarnival.com