Quick poll Welcome to sunny London! Tammer Saleh Geek: Unix, Ruby, - - PowerPoint PPT Presentation

▶

quick poll welcome to sunny london tammer saleh

Quick poll Welcome to sunny London! Tammer Saleh Geek: Unix, Ruby, - - PowerPoint PPT Presentation

Dec 30, 2022 318 likes •1.1k views

Quick poll Welcome to sunny London! Tammer Saleh Geek: Unix, Ruby, Golang , etc Cloud Foundry @ Pivotal http://tammersaleh.com | tsaleh@pivotal.io Microservice Anti-patterns How not to go down in flames. Why microservices? What is a

slide-1

SLIDE 1

Quick poll…

slide-2

SLIDE 2

Welcome to sunny London!

slide-3

SLIDE 3

Tammer Saleh

Geek: Unix, Ruby, Golang, etc Cloud Foundry @ Pivotal http://tammersaleh.com | tsaleh@pivotal.io

slide-4

SLIDE 4

Microservice Anti-patterns

How not to go down in flames.

slide-5

SLIDE 5

Why microservices?

What is a microservice, and why do I care?

slide-6

SLIDE 6

Monolithic

Entire application in a single codebase, deployed and scaled as a single unit.

slide-7

SLIDE 7

Monolithic

Hard to scale the application. Impossible to scale the team.

slide-8

SLIDE 8

It’s not about code… It’s about teams.

slide-9

SLIDE 9

But it can go wrong.

Here are the most common problems we see in the wild, and how to fix them.

slide-10

SLIDE 10

Overzealous Services

The most common mistake is to

start with microservices

slide-11

SLIDE 11

Boring is Beautiful™

slide-12

SLIDE 12

Solution: Start monolithic and extract

Microservices are complex and add a constant tax to development. Build a boring application and extract services as needed.

slide-13

SLIDE 13

Twitter

/tweet /dashboard /… /… Heavy and Expensive

slide-14

SLIDE 14

/tweet

Twitter

/dashboard /… /… Lite and Cheap

slide-15

SLIDE 15

Congratulations

You’re now a microservice architect.

slide-16

SLIDE 16

Schemas everywhere

/dashboard /… /… /tweet

v1 v1

V1 Schema

Deploy v2

slide-17

SLIDE 17

Schemas everywhere

/dashboard /… /… /tweet

v1 v2

V1 Schema

Deploy v2 Migrate v2

slide-18

SLIDE 18

/dashboard /… /… /tweet

v1 v2

V2 Schema

Deploy v2

Schemas everywhere

Migrate v2

slide-19

SLIDE 19

/dashboard /… /… /tweet

v2 v2

V2 Schema

Schemas everywhere

slide-20

SLIDE 20

Solution: Gatekeeper

/dashboard /… /… /tweet

v1 v2

Tweets

/bulk_add

Tweet Service

GET /tweets GET /tweets/ID Owns database and migrations

slide-21

SLIDE 21

Lock-step deployment

Service B Service A Service C — v1

v1 v1

slide-22

SLIDE 22

Lock-step deployment

Service B Service A Service C — v2

v1 v1 Deploy v2

slide-23

SLIDE 23

Lock-step deployment

Service B Service A Service C — v2

v1 v1 Deploy v2

slide-24

SLIDE 24

Lock-step deployment

Service B Service A Service C — v2

v2 v2 Deploy v2

slide-25

SLIDE 25

Solution: Semantic Versioning

Service B Service A Service C — v1

v1 v1

vMajor.Minor.Patch MYBAD.SHINY.OOPS

slide-26

SLIDE 26

Solution: Semantic Versioning

Service B Service A Service C — v1.2

v1 v1 Deploy v1.2 (extra stuff)

:) :)

slide-27

SLIDE 27

Solution: Semantic Versioning

Service B Service A Service C — v1.2

v1.2 v1.2 Deploy v1.2

slide-28

SLIDE 28

Solution: Semantic Versioning

Service B Service A Service C — v1.2

v1.2 v1.2 Deploy v2 (breaking change)

Service C — v2

slide-29

SLIDE 29

Solution: Semantic Versioning

Service B Service A Service C — v1.2

v1.2 v2

Service C — v2

Deploy v2

slide-30

SLIDE 30

Solution: Semantic Versioning

Service B Service A Service C — v1.2

v2 v2

Service C — v2

Deploy v2

slide-31

SLIDE 31

OMG ALL THE STEPS!!!

See Rule #1

slide-32

SLIDE 32

Spiky load between services

Must maintain enough servers to handle peak load.

slide-33

SLIDE 33

/tweet /dashboard /… /…

Spiky load between services

slide-34

SLIDE 34

/tweet /dashboard /… /…

Spiky load between services

slide-35

SLIDE 35

Queues in between services provide buffers that smooth traffic.

Solution: Amortize via queues

Peak load is now much lower.

slide-36

SLIDE 36

/tweet /dashboard /… /…

Solution: Amortize via queues

slide-37

SLIDE 37

Solution: Amortize via queues

/tweet /dashboard /… /… Worker

Complexity: Now clients must deal with asynchronous responses.

slide-38

SLIDE 38

Hardcoded IPs and Ports

Simple to get started, but immediately leads to deployment issues.

slide-39

SLIDE 39

Solution 1: Discovery Service

etcd consul

slide-40

SLIDE 40

Solution 1: Discovery Service

Service A Service B (192.168.0.2:1234)

Where is B?

192.168.0.2:1234

slide-41

SLIDE 41

Solution 1: Discovery Service

Service A Service B (192.168.0.2:1234)

Hey there!

slide-42

SLIDE 42

Solution 1: Discovery Service

Complexity: Your code must understand the service lookup system.

slide-43

SLIDE 43

Solution 2: Centralised router

Service A Service B Router Service B Service A Service A Service B

A A B B A A B B A A B B

slide-44

SLIDE 44

Solution 2: Centralised router

Simplicity: “It’s just DNS.”

slide-45

SLIDE 45

Router vs Discovery Service

Both require service registration. Discovery Service is simpler to build and scale, since it doesn’t need to route all data. Discovery service does fewer network hops. Router is transparent. Router can be exposed externally. Router can cache transparently. Router can round-robin. Both require HA and scalability.

slide-46

SLIDE 46

Dogpiles

Service B Service A

You there? No. How about now? I’m busy. But I wanna chat! Go away Whachya doin? Please leave me alone. Wanna be friends? I hate you.

slide-47

SLIDE 47

Dogpiles

Service B Service A Service A Service A Service A

x 100

slide-48

SLIDE 48

Solution: Circuit Breaker

Service B Service A Service A

Where’s B? B’s sick. Wait 10 min. Hello! Go to Hell! Hey, B’s sick :(

slide-49

SLIDE 49

Debugging hell

Turns out, distributed systems are hard.

slide-50

SLIDE 50

Debugging Hell

Service 3 Service 1 Service 2

time:1428374783 source:service3 msg:”Error with payment!”

Where?? Why?? Who??

slide-51

SLIDE 51

Solution: Correlation IDs

Service Service Service

CID: 1234 CID: 1234 CID: 1234 time:1427127483 source:service1 id:1234 msg:”Received request…” time:1427348748 source:service2 id:1234 msg:”Processing payment" time:1428374783 source:service3 id:1234 msg:”Error with payment!”

slide-52

SLIDE 52

Solution: Correlation IDs

1. Tag all incoming requests with

unique ID

2. Service saves ID for all incoming

requests

3. Include that ID in all log lines,

etc.

4. Tag new requests with that ID

Service

Logs

ID

ID ID ID

Complexity: Must be done manually.

slide-53

SLIDE 53

Missing Mock Servers

Each consuming team has to create their own mocks and stubs.

slide-54

SLIDE 54

Missing Mock Servers

Service A

Team A

slide-55

SLIDE 55

Missing Mock Servers

Service A Service B

HTTP

Team A Team B

Product Surface Area

slide-56

SLIDE 56

Missing Mock Servers

Service A Service B

HTTP

Mock of Service A

HTTP

Team A Team B For each service… For each team…

slide-57

SLIDE 57

Service A Service B

HTTP

Mock of Service A

HTTP

Team A Team B Better…

Solution: Service Team Provides the Mock

Team B still needs to know how to run Mock Service A

slide-58

SLIDE 58

Solution: Service Team Owns the Client

Service A Service B

HTTP H T T P

Team A Team B

Client A

MOCK=“true”

Best…

Mock of Service A Product Surface Area Both modes are tested in CI T eam A can change the protocol as they see fit

slide-59

SLIDE 59

Flying Blind

slide-60

SLIDE 60

Solution: Graphs, alerts, pages.

slide-61

SLIDE 61

Solution: Graphs, alerts, pages.

Alerting Rules Engine Platform Events

O M G W A T ? ? ?

Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps

slide-62

SLIDE 62

Snowflakes

slide-63

SLIDE 63

Snowflakes

Ruby XSS exploit! BASH remote exploit!

slide-64

SLIDE 64

Snowflakes

Containers Virtual Machines

slide-65

SLIDE 65

Solution: Golden Image

slide-66

SLIDE 66

Solution: Golden Image

Golden OS Image Common Runtimes and Frameworks

+

Base Platform

=

slide-67

SLIDE 67

Doomsday Deployments

slide-68

SLIDE 68

Solution: Predictable Pipelines

slide-69

SLIDE 69

Solution: Predictable Pipelines

http://concourse.ci

slide-70

SLIDE 70

Solution: Predictable Pipelines Need to trust your tests, your platform, and your automation.

slide-71

SLIDE 71

Operational Explosion!

slide-72

SLIDE 72

Operational Explosion!

Operations block Development

slide-73

SLIDE 73

Solution: AUTOMATE ALL THE THINGS!!!!!

Form a team to build tools that enable developers to manage the system in an entirely automated way.

slide-74

SLIDE 74

OMG ALLTHETHINGS????

App deployment, infrastructure provisioning, OS installation,

configuration management, database provisioning, disaster recovery, application monitoring, HA, blue-green deployments,

self-healing, scaling, runtime installation, log rotation, backups, security updates, database upgrades, application logs, system logs, database logs, continuous integration, continuous deployment, service discovery,

monitor queue usage, security monitoring, hotspot detection, error monitoring, issue notification and escalation, virtual machine migration, shard rebalancing,

circuit breaker monitoring, resiliency testing, database snapshots, flux capacitors,

ion overdrive maintenance, change the oil, dog feeding, cat shooting, pig eating…

slide-75

SLIDE 75

Time and Money

Solution: AUTOMATE ALL THE THINGS!!!!!

slide-76

SLIDE 76

slide-77

SLIDE 77

In summary…

Start boring and extract to services. Understand the hidden schemas. Amortize traffic with queues. Decouple through discovery tools. Contain failures with circuit breakers. Enable other teams through mockable clients. Kill your snowflakes. Automate your deployments. Build in operations tools from the beginning.

Make use of a platform like Cloud Foundry.