How to build observable distributed systems @PierreVincent - - PowerPoint PPT Presentation

▶

Mar 15, 2023 183 likes •538 views

March 8th, 2018 QCon London How to build observable distributed systems @PierreVincent pvincent.io @PierreVincent @PierreVincent Pierre Vincent SRE Manager at Poppulo @PierreVincent pvincent.io @PierreVincent Reaching production is

SLIDE 1

@PierreVincent

How to build observable distributed systems

March 8th, 2018 – QCon London @PierreVincent pvincent.io

SLIDE 2

@PierreVincent

SLIDE 3

@PierreVincent

Pierre Vincent

SRE Manager at Poppulo

@PierreVincent pvincent.io

SLIDE 4

@PierreVincent

Reaching production is

nly the beginning

SLIDE 5

@PierreVincent

No system is immune to failure Be ready to recover

SLIDE 6

@PierreVincent

When distributing a system, we’re also distributing the places where things might go wrong

SLIDE 7

@PierreVincent

Monitoring only applies to known failure modes. What about everything else?

SLIDE 8

@PierreVincent

Monitoring tells you whether the system works. Observability lets you ask why it's not working.

“ ”

– Baron Schwarz

SLIDE 9

@PierreVincent

Healthchecks Metrics Logging Tracing

SLIDE 10

@PierreVincent

Healthchecks

SLIDE 11

@PierreVincent

Is it running? Can it perform its task? Can it accept more work?

Healthchecks

SLIDE 12

@PierreVincent

Broadcast Register Expose

Healthchecks

SLIDE 13

@PierreVincent

Healthchecks

/health { "service": "registration-service", "healthy": true, "workload": { "healthy": true }, "dependencies": [ { "name": "cassandra", "healthy": true }, { "name": "billing-svc", "healthy": true }, ] } GET http://1.2.3.4:8080/health 200 OK

SLIDE 14

@PierreVincent

Source: HTTP Healthchecks for a Resilient Platform - Chris O’Dell

skeltonthatcher.com/blog/http-healthchecks-for-a-resilient-platform

Overzealous Healthchecks can be counter-productive

SLIDE 15

@PierreVincent

Metrics

SLIDE 16

@PierreVincent

System metrics Application metrics Business metrics

CPU usage Error rates Customer conversions

Metrics

SLIDE 17

@PierreVincent Servers / VMs Appliances/Infra Services Metrics collector Metrics query engine

Dashboards Alerts

Metrics

SLIDE 18

@PierreVincent Servers / VMs Appliances/Infra Services

/metrics /metrics /metrics

Prometheus

Metrics

SLIDE 19

@PierreVincent

Watch out for over-reliance on metrics

Limitations at high-cardinality Not every metric deserves an alert Real-time querying means some trade-offs on retention Limit alerting to user-impacting symptoms Poor fine-grained debugging e.g. CustomerId Not suitable for long-term trend analysis

SLIDE 20

@PierreVincent

Logging

SLIDE 21

@PierreVincent

Searchable Correlated

Logging

Making sense of (a lot of) logs

Centralised

SLIDE 22

@PierreVincent

A F H D J B E C G

a1b2c3 a1b2c3 a1b2c3

ERROR [svc=H][trace=a1b2c3] Failed to save order Cause: Cassandra timeout exception ERROR [svc=F][trace=a1b2c3] Failed to complete order Cause: Shipping service responded with 500 ERROR [svc=A][trace=a1b2c3] Failed to process order Cause: Order process manager responded with 500

a1b2c3

INFO [svc=G][trace=a1b2c3] Items verified in stock

Log Correlation

SLIDE 23

@PierreVincent

JSON

2018-02-20T16:38:23+00:00 ERROR Read timed out

timestamp 2018-02-20T16:38:23+00:00 requestID ec667cb45 level ERROR team events service registration-service commit 542a8b8e build 542a8b8e.7 node node_e79f3e52 log Read timed out region europe-west2 runtime java-1.8.0_161 When did it happen? Can I trace it? What is it? What is it running? Where is it? What is the message? customerID 55123 Who caused it? userID 458 ... ... Any other info? Hmmm thanks… ?

SLIDE 24

@PierreVincent

Structured logs unleash high-cardinality exploration

Error rate spike isolated by build version

Activity spike isolated for single customer

SLIDE 25

@PierreVincent

Tracing

SLIDE 26

@PierreVincent

Tracing

get_confirmed_attendees get_attendees get_confirm_status cassandra/select mysql/select event-mgt-api attendees-service registration-service

Trace Spans

0ms 50ms 100ms 150ms 172ms 172ms 73ms 54ms 78ms 41ms

SLIDE 27

@PierreVincent

SLIDE 28

@PierreVincent

Unknown Unknowns Known Unknowns Healthchecks Metrics

(Alerting)

Logs Metrics

(Queries)

Tracing Events

Monitoring & Resiliency Debugging & Exploration

SLIDE 29

@PierreVincent

Cheap to instrument Easy to explore Reliable & Trustworthy

Usability of tooling is key to adoption

SLIDE 30

@PierreVincent

Visibility builds trust but requires safety

SLIDE 31

@PierreVincent

Visibility helps justify decisions

SLIDE 32

@PierreVincent

Visibility enables operability

SLIDE 33

@PierreVincent

Test (a little bit*) less Test (a little bit*) less

Don’t spend all your time testing

...keep some for instrumenting

Thank you!

@PierreVincent pvincent.io

SLIDE 34

@PierreVincent

Thank you!

Pierre Vincent

SRE Manager at Poppulo

@PierreVincent pvincent.io