Goutham Veeramachaneni @putadent Me Goutham Veeramachaneni | 2 - - PowerPoint PPT Presentation

▶
goutham veeramachaneni
SMART_READER_LITE
LIVE PREVIEW

Goutham Veeramachaneni @putadent Me Goutham Veeramachaneni | 2 - - PowerPoint PPT Presentation

Feb 18, 2023 417 likes •697 views

Goutham Veeramachaneni @putadent Me Goutham Veeramachaneni | 2 Debugging slow queries Cortex | 3 | 4 A few large queries are slow | 5 RED Method | 6 Soooo whats slow? Hypothesis Experiment Metrics | 7 Saving my arse since June 2018 | 8

slide-1
SLIDE 1

Goutham Veeramachaneni

@putadent

slide-2
SLIDE 2

Me

Goutham Veeramachaneni | 2

slide-3
SLIDE 3

Debugging slow queries

Cortex | 3

slide-4
SLIDE 4

| 4

slide-5
SLIDE 5

A few large queries are slow

| 5

slide-6
SLIDE 6

RED Method

| 6

slide-7
SLIDE 7

Soooo whats slow?

| 7

Hypothesis Metrics Experiment

slide-8
SLIDE 8

Saving my arse since June 2018

| 8

slide-9
SLIDE 9

Soooo whats slow?

| 9

slide-10
SLIDE 10

Soooo whats slow?

| 10

slide-11
SLIDE 11

| 11

slide-12
SLIDE 12

Is Jaeger the solution to everything?

| 12

slide-13
SLIDE 13

My new flow

| 13

slide-14
SLIDE 14

Per environment traces?

| 14

  • Several clusters but central jaeger
slide-15
SLIDE 15

Enriching traces

| 15

  • blog
slide-16
SLIDE 16

Enriching traces

| 16

  • Just add namespace and clustername, use env vars

jaeger_mixin:: if $._config.jaeger_agent_host == null then {} else container.withEnvMixin([ container.envType.new('JAEGER_AGENT_HOST', $._config.jaeger_agent_host), container.envType.new('JAEGER_TAGS', 'namespace=%s,cluster=%s' % [$._config.namespace, $._config.cluster]), container.envType.new('JAEGER_SAMPLER_MANAGER_HOST_PORT', 'http://%s:5778/sampling' % $._config.jaeger_agent_host), ]),

slide-17
SLIDE 17

Per environment traces?

  • Several clusters but central jaeger

| 17

Alerts Dashboard Jaeger

slide-18
SLIDE 18

Monitoring Jaeger with Prometheus

| 18

slide-19
SLIDE 19

We’re only scratching the surface

| 19

slide-20
SLIDE 20

Exemplars: Seamless flow

| 20

slide-21
SLIDE 21

Exemplars: Seamless flow

| 21

slide-22
SLIDE 22

Exemplars: Seamless flow

| 22

slide-23
SLIDE 23

More future

| 23

  • Grafana has tracing built in (Jaeger being first)
  • Integrate the logs and traces and metrics
  • Built in UI for visualising Prometheus queries
  • Conprof
slide-24
SLIDE 24

Cortex | 24

slide-25
SLIDE 25

Recap

| 25

  • Jaeger is awesome for debugging latency
  • Alerts —> Dashboard —> Jaeger, make sure the flow works
  • The jaeger_mixin helps you tame Jaeger
  • Exemplars are coming!
slide-26
SLIDE 26

@putadent