Help! I accidentally distributed my system! Rachel Myers, Emily - - PowerPoint PPT Presentation

▶
help i accidentally distributed my system
SMART_READER_LITE
LIVE PREVIEW

Help! I accidentally distributed my system! Rachel Myers, Emily - - PowerPoint PPT Presentation

Mar 25, 2024 221 likes •1.2k views

Help! I accidentally distributed my system! Rachel Myers, Emily Nakashima @eanakashima @rachelmyers, @eanakashima @rachelmyers, @eanakashima @rachelmyers @rachelmyers, @eanakashima @rachelmyers, @eanakashima @rachelmyers, @eanakashima

slide-1
SLIDE 1

Help! I accidentally distributed my system!

Rachel Myers, Emily Nakashima

slide-2
SLIDE 2 @rachelmyers, @eanakashima

@eanakashima

slide-3
SLIDE 3 @rachelmyers, @eanakashima
slide-4
SLIDE 4 @rachelmyers, @eanakashima

@rachelmyers

slide-5
SLIDE 5 @rachelmyers, @eanakashima
slide-6
SLIDE 6 @rachelmyers, @eanakashima
slide-7
SLIDE 7 @rachelmyers, @eanakashima ModCloth

Web App
 (Rails) MySQL

User request (from NGINX & loadbalancer)
slide-8
SLIDE 8 @rachelmyers, @eanakashima

What even is
 a distributed system?

slide-9
SLIDE 9 @rachelmyers, @eanakashima What even is a distributed system?

Some
 Node Another Node

slide-10
SLIDE 10 @rachelmyers, @eanakashima

Designing distributed systems

  • What are the “nouns” in my system?
  • What are the patterns for reading data?
  • What are the patterns for writing data?
  • Do different flows have different performance, consistency,

& availability requirements?
 


👓

slide-11
SLIDE 11 @rachelmyers, @eanakashima ModCloth

Web App MySQL

User request (from NGINX & loadbalancer)

Mobile

slide-12
SLIDE 12 @rachelmyers, @eanakashima ModCloth

Web App MySQL

User request (from NGINX & loadbalancer)

Mobile

Voting Service

MySQL
slide-13
SLIDE 13 @rachelmyers, @eanakashima ModCloth

Web App MySQL Identity Service

User request (from NGINX & loadbalancer)

Mobile

MySQL

Voting Service

MySQL
slide-14
SLIDE 14 @rachelmyers, @eanakashima ModCloth

Web App MySQL Identity Service

User request (from NGINX & loadbalancer)

Mobile

MySQL

Voting Service Asset Service

MySQL MySQL
slide-15
SLIDE 15 @rachelmyers, @eanakashima ModCloth

Web App MySQL Identity Service Photo Service

User request (from NGINX & loadbalancer)

Mobile

MySQL

Voting Service Asset Service Comment Service

MySQL MySQL MySQL MySQL
slide-16
SLIDE 16 @rachelmyers, @eanakashima ModCloth

Web App MySQL Identity Service Photo Service Moderation Service

User request (from NGINX & loadbalancer)

Mobile

MySQL

Voting Service Asset Service Comment Service

MySQL MySQL MySQL MySQL MySQL
slide-17
SLIDE 17 @rachelmyers, @eanakashima ModCloth

Web App MySQL Checkout Service Identity Service Photo Service Moderation Service

User request (from NGINX & loadbalancer)

Mobile

MySQL

Voting Service Asset Service Comment Service

MySQL MySQL MySQL MySQL MySQL MySQL
slide-18
SLIDE 18 @rachelmyers, @eanakashima ModCloth

Web App MySQL Checkout Service Identity Service Photo Service Moderation Service

User request (from NGINX & loadbalancer)

Mobile

MySQL

Voting Service Asset Service Comment Service

MySQL MySQL MySQL MySQL MySQL MySQL 🔦

🗒

slide-19
SLIDE 19 @rachelmyers, @eanakashima ModCloth

Web App MySQL Checkout Service Identity Service Photo Service Moderation Service

User request (from NGINX & loadbalancer)

Mobile

MySQL

Voting Service Asset Service Comment Service

MySQL MySQL MySQL MySQL MySQL MySQL

😈😈😈 🤔🤔🤔

slide-20
SLIDE 20 @rachelmyers, @eanakashima

And this was intentional 😃

slide-21
SLIDE 21 @rachelmyers, @eanakashima

An accidental distributed system ⛈

flexfits.com
slide-22
SLIDE 22 @rachelmyers, @eanakashima flexfits.com
slide-23
SLIDE 23 @rachelmyers, @eanakashima flexfits.com

Ecommerce SaaS

User request

slide-24
SLIDE 24 @rachelmyers, @eanakashima flexfits.com

Subscription Plugin SaaS

Ecommerce SaaS

User request

slide-25
SLIDE 25 @rachelmyers, @eanakashima flexfits.com

Subscription Plugin SaaS

Ecommerce SaaS Fulfillment SaaS

User request

slide-26
SLIDE 26 @rachelmyers, @eanakashima flexfits.com

Subscription Plugin SaaS

Ecommerce SaaS Metrics SaaS Fulfillment SaaS

User request

slide-27
SLIDE 27 @rachelmyers, @eanakashima flexfits.com
slide-28
SLIDE 28 @rachelmyers, @eanakashima flexfits.com

👒 👒

slide-29
SLIDE 29 @rachelmyers, @eanakashima flexfits.com

Subscription Plugin SaaS

Ecommerce SaaS Metrics SaaS Fulfillment SaaS

User request

?

slide-30
SLIDE 30 @rachelmyers, @eanakashima flexfits.com

Subscription Plugin SaaS

Ecommerce SaaS Metrics SaaS Fulfillment SaaS

User request

Orders Service

Postgres

slide-31
SLIDE 31 @rachelmyers, @eanakashima flexfits.com

Subscription Plugin SaaS

Ecommerce SaaS Metrics SaaS Fulfillment SaaS

User request

Orders Service

Postgres

👓

slide-32
SLIDE 32 @rachelmyers, @eanakashima

New requirements for buy vs build

slide-33
SLIDE 33 @rachelmyers, @eanakashima flexfits.com

SaaS

SaaS SaaS SaaS

User request

Home cooked

  • n IaaS
slide-34
SLIDE 34 @rachelmyers, @eanakashima

Buy or Build

  • Will it improve availability?
  • Will it improve recovery?
  • Will you have visibility or points of control?
  • Will you have customizability?
  • Will you know when your changes take effect?
  • Can you audit changes?
slide-35
SLIDE 35 @rachelmyers, @eanakashima

IaaS, PaaS, BaaS, & FaaS

slide-36
SLIDE 36 @rachelmyers, @eanakashima

Cloud Infrastructure

slide-37
SLIDE 37 @rachelmyers, @eanakashima

IaaS

slide-38
SLIDE 38 @rachelmyers, @eanakashima

IaaS

  • Will it improve availability? ✅
  • Will it improve recovery? ✅
  • Will you have visibility or points of control? ✅
  • Will you have customizability? ✅
  • Will you know when your changes take effect? ✅
  • Can you audit changes? ✅
slide-39
SLIDE 39 @rachelmyers, @eanakashima

Platforms

slide-40
SLIDE 40 @rachelmyers, @eanakashima

PaaS

slide-41
SLIDE 41 @rachelmyers, @eanakashima

PaaS

  • Will it improve availability? ✅
  • Will it improve recovery? 🤕
  • Will you have visibility or points of control? ✅
  • Will you have customizability? ✅
  • Will you know when your changes take effect? 🤕
  • Can you audit changes? 🤕
slide-42
SLIDE 42 @rachelmyers, @eanakashima

Backends

slide-43
SLIDE 43 @rachelmyers, @eanakashima

BaaS

(RIP Parse)
slide-44
SLIDE 44 @rachelmyers, @eanakashima

BaaS

  • Will it improve availability? ✅🤕
  • Will it improve recovery? 🤕
  • Will you have visibility or points of control? 🤕
  • Will you have customizability? ✅
  • Will you know when your changes take effect? 🤕
  • Can you audit changes? 🤕
slide-45
SLIDE 45 @rachelmyers, @eanakashima

Functions

slide-46
SLIDE 46 @rachelmyers, @eanakashima

FaaS

slide-47
SLIDE 47 @rachelmyers, @eanakashima

FaaS

  • Will it improve availability? 🤕
  • Will it improve recovery? 🤕
  • Will you have visibility or points of control? 🤕
  • Will you have customizability? ✅
  • Will you know when your changes take effect? 🤕
  • Can you audit changes? 🤕
slide-48
SLIDE 48

How to choose?

slide-49
SLIDE 49 @rachelmyers, @eanakashima

Infrastructure

Platform

Backend

Faster to start Harder to change

slide-50
SLIDE 50 @rachelmyers, @eanakashima
slide-51
SLIDE 51 @rachelmyers, @eanakashima
slide-52
SLIDE 52 @rachelmyers, @eanakashima

Surprise Distributed Systems

slide-53
SLIDE 53 @rachelmyers, @eanakashima

Surprise distributed system: browsers

slide-54
SLIDE 54 @rachelmyers, @eanakashima

“The browser is part of your distributed system, not just a client you support”


  • @danielespeset
slide-55
SLIDE 55 @rachelmyers, @eanakashima

The one where we DDOS’d ourselves

slide-56
SLIDE 56 @rachelmyers, @eanakashima

What happened?

slide-57
SLIDE 57 @rachelmyers, @eanakashima // if the page is long enough to scroll if (document.body.clientHeight > window.innerHeight) { // add a scroll event listener document.addEventListener('scroll', function(e) { // if within 100px of the bottom of the page if (window.innerHeight + window.scrollY + 100 > document.body.clientHeight) fetchNextPage(); }); // else fetch another page of results immediately } else { fetchNextPage(); }
slide-58
SLIDE 58 @rachelmyers, @eanakashima

Frontend complexity is only increasing

slide-59
SLIDE 59 @rachelmyers, @eanakashima

Instrumentation Rule 1: Start at the edge.
 Corollary: The edge may be further than you think.

slide-60
SLIDE 60 @rachelmyers, @eanakashima

Secret distributed system: vendors

slide-61
SLIDE 61 @rachelmyers, @eanakashima

LaunchDarkly

slide-62
SLIDE 62 @rachelmyers, @eanakashima

Secret distributed system: vendors

slide-63
SLIDE 63 @rachelmyers, @eanakashima # instantiate the flag with a default value # falls back to default value if service is offline show_feature = ld_client.variation( "allow_traces", { :key => "user@example.com" }, false ) if show_feature # application code to show the feature else # the code to run if the feature is off
slide-64
SLIDE 64 @rachelmyers, @eanakashima

Secret distributed system: vendors

slide-65
SLIDE 65 @rachelmyers, @eanakashima

Instrumentation Rule 2:

Wrap 3rd party calls

slide-66
SLIDE 66 @rachelmyers, @eanakashima

Secret distributed system: vendors

slide-67
SLIDE 67 @rachelmyers, @eanakashima

GitHub’s customer support application

slide-68
SLIDE 68 @rachelmyers, @eanakashima GitHub customer support tool

github.com

Customer Support Tool

User email

slide-69
SLIDE 69 @rachelmyers, @eanakashima GitHub customer support tool

github.com

Customer Support Tool GMail

User email

slide-70
SLIDE 70 @rachelmyers, @eanakashima GitHub customer support tool

github.com

MailChimp Customer Support Tool GMail

User email

SendGrid

slide-71
SLIDE 71 @rachelmyers, @eanakashima GitHub customer support tool

github.com

MailChimp Customer Support Tool GMail

User email

SendGrid

slide-72
SLIDE 72 @rachelmyers, @eanakashima GitHub customer support tool

github.com

Kafka Customer Support Tool Ingest service

User email

Outbound mail service

slide-73
SLIDE 73 @rachelmyers, @eanakashima GitHub customer support tool

github.com

Kafka Customer Support Tool Ingest service

User email

Outbound mail service 🔦

🗒

slide-74
SLIDE 74 @rachelmyers, @eanakashima GitHub customer support tool

github.com

MailChimp Customer Support Tool GMail

User email

SendGrid

slide-75
SLIDE 75 @rachelmyers, @eanakashima

Cognitive Load

slide-76
SLIDE 76 @rachelmyers, @eanakashima

Instrumentation & Chill ❄

slide-77
SLIDE 77 @rachelmyers, @eanakashima

Fewer Alerts 📠

slide-78
SLIDE 78 @rachelmyers, @eanakashima
slide-79
SLIDE 79 @rachelmyers, @eanakashima

Unknown unknown unknowns 🔦

slide-80
SLIDE 80 @rachelmyers, @eanakashima

Outsource Your Brain 📙

slide-81
SLIDE 81 @rachelmyers, @eanakashima

Use tools that let you…

  • Ask, what is happening now to this user? Endpoint? Team?
  • Ask, how does this compare to the aggregate case?
  • Jump between levels of abstraction
  • See both breadth & depth
  • Understand how data flows

slide-82
SLIDE 82 @rachelmyers, @eanakashima

Observability… not just for

  • ps thought leaders!
slide-83
SLIDE 83 @rachelmyers, @eanakashima

“The ability to ask arbitrary questions about your system from the outside, and understand what is happening on the inside.”


  • @mipsytipsy
slide-84
SLIDE 84 @rachelmyers, @eanakashima

logs metrics

traces / events

slide-85
SLIDE 85 @rachelmyers, @eanakashima

How to pick an

  • bservability tool?
slide-86
SLIDE 86 @rachelmyers, @eanakashima

Crawl the pipes

slide-87
SLIDE 87 @rachelmyers, @eanakashima flexfits.com
slide-88
SLIDE 88 @rachelmyers, @eanakashima

No really, instrumentation

slide-89
SLIDE 89 @rachelmyers, @eanakashima
slide-90
SLIDE 90 @rachelmyers, @eanakashima # monitoring endpoint Location ~ /ping { return 200 '200 ok'; }
slide-91
SLIDE 91 @rachelmyers, @eanakashima

Instrumentation Rule 3:

Your instrumentation should support depth- *and* breadth-first

slide-92
SLIDE 92 @rachelmyers, @eanakashima

Conclusion

slide-93
SLIDE 93 @rachelmyers, @eanakashima

Two Vendors: Two Conclusions 😭

slide-94
SLIDE 94 @rachelmyers, @eanakashima
  • 1. We’re all distributed

systems engineers

  • perators.

☎ 🐣

slide-95
SLIDE 95 @rachelmyers, @eanakashima
  • 2. Trust your tools

 🛡

slide-96
SLIDE 96 @rachelmyers, @eanakashima

Thank you