Introduction to Distributed Tracing Joe Elliott, Annanay Agarwal - - PowerPoint PPT Presentation

▶
introduction to distributed tracing
SMART_READER_LITE
LIVE PREVIEW

Introduction to Distributed Tracing Joe Elliott, Annanay Agarwal - - PowerPoint PPT Presentation

Sep 28, 2022 260 likes •455 views

Introduction to Distributed Tracing Joe Elliott, Annanay Agarwal What are we doing here? - Intro! - Jaeger Getting Started - OpenTracing: What is Tracing - OpenTracing Examples - Open Source - Demo - Jaeger All In One +

slide-1
SLIDE 1

Introduction to Distributed Tracing

Joe Elliott, Annanay Agarwal

slide-2
SLIDE 2
  • Intro!
  • Jaeger Getting Started
  • OpenTracing: What is Tracing
  • OpenTracing Examples
  • Open Source
  • Demo
  • Jaeger All In One + Instrumentation
  • Grafana + Jaeger Integrations

What are we doing here?

slide-3
SLIDE 3

Why?

slide-4
SLIDE 4

Images: Prometheus, Influx, Loki, Elastic

Monitoring tools

Logs Metrics

slide-5
SLIDE 5

https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/

RED Method

Requests Error Duration

USE Method

Utilization Saturation Errors

….

Metrics - Aggregatable

slide-6
SLIDE 6

http_request_duration_sec{“app=ice-cream-shop”} 10s

Metrics - Cardinality

slide-7
SLIDE 7

http_request_duration_sec{“app=ice-cream-shop” datacenter=”us-central”, env=”production”, service=”cart-manager”, path=”/api/order”, func_name=”my-func”, cust_name=”annanay”} 10s

Metrics - Cardinality

slide-8
SLIDE 8

Img src: https://www.dictionary.com/e/emoji/heavy-dollar-sign-emoji/

http_request_duration_sec{“app=ice-cream-shop” datacenter=”us-central”, env=”production”, service=”cart-manager”, path=”/api/order”, func_name=”my-func”, cust_name=”annanay”} 10s

Metrics - Cardinality

🐍

slide-9
SLIDE 9

Logs

slide-10
SLIDE 10

Useful to check health of a particular service. “Events” in a service.

"rpc error: code = Code(400) desc = user = xyz: series={__name__=\"process_virtual_memory_bytes\", cluster=\"us-central\", instance=\"consul-5sqwxccxvq\", timestamp=2020-01-04T14:12:59.102Z: out of order sample}"

. .

level=error ts=2020-05-08T09:46:43.11193994Z caller=a_file_from_my_codebase:16 msg="error processing requests" err="rpc error: Code(400) desc = expected string but got null"

Logs

slide-11
SLIDE 11

TLDR;

Telemetry Primary use case What we want in addition Metrics Aggregations Fine grained information Logs Events Cross process tracing

slide-12
SLIDE 12

Distributed Tracing

slide-13
SLIDE 13

Would You Like Some Tracing With Your Monitoring? - Yuri Shkuro, Uber Technologies - KCCNA 2017

Diagnosing the Long Tail

slide-14
SLIDE 14

Inspired from - Would You Like Some Tracing With Your Monitoring? - Yuri Shkuro, Uber Technologies - KCCNA 2017

Distributed Tracing!

slide-15
SLIDE 15

https://www.jaegertracing.io/docs/1.17/architecture/

How? Context propagation

slide-16
SLIDE 16
  • Context Propagation (OpenTracing Nethttp)
  • In Process : Context
  • Cross Process : Headers
  • Metadata
  • Logs, Span Tags, Process Tags
  • Explore the Jaeger UI
  • Grafana/Loki Trace Integration

https://github.com/joe-elliott/tracing-example

Demo! (Jaeger + OpenTracing)

slide-17
SLIDE 17

Getting involved

slide-18
SLIDE 18

Open Source

slide-19
SLIDE 19

Thank you for watching!

Have more questions? Join us at community.grafana.com.

LIVE WEBINAR Introduction To Distributed Tracing | with Joe Elliott