Bomb Squad Containing the Cardinality Explosion Cody Boggs PromCon - - PowerPoint PPT Presentation

▶
bomb squad
SMART_READER_LITE
LIVE PREVIEW

Bomb Squad Containing the Cardinality Explosion Cody Boggs PromCon - - PowerPoint PPT Presentation

Sep 04, 2023 439 likes •593 views

Bomb Squad Containing the Cardinality Explosion Cody Boggs PromCon 2018, Munich @strofcon Who? Cody Chowny Boggs Ops Nerd of ~8 years Lead yak shaver @ FreshTracks for ~1 year Obsessed with metrics cody@freshtracks.io

slide-1
SLIDE 1

@strofcon

Bomb Squad

Containing the Cardinality Explosion

PromCon 2018, Munich

Cody Boggs

slide-2
SLIDE 2

@strofcon

Who?

Cody “Chowny” Boggs

  • Ops Nerd of ~8 years
  • Lead yak shaver @ FreshTracks for

~1 year

  • Obsessed with metrics
  • Pretends to write code
  • Breaks things. (All the things.)

cody@freshtracks.io

slide-3
SLIDE 3

@strofcon

On Deck

  • What is cardinality?
  • What is a “cardinality explosion”?
  • Who cares?
  • Charts and graphs!
  • Bomb Squad live demo!
slide-4
SLIDE 4

@strofcon

What is cardinality...

Generally? The number of elements in a set or group { b, 42, tree} Cardinality: 3

Images: https://openclipart.org/detail/133471/cardinal-remix-1

slide-5
SLIDE 5

@strofcon

What is cardinality...

For this talk? The number of discrete label/value pairs (series) associated with a particular metric cpu{host=”foo”} cpu{host=”bar”} cpu{host=”broken”} Cardinality: 3

Images: https://openclipart.org/detail/133471/cardinal-remix-1

slide-6
SLIDE 6

@strofcon

Words that mean things!

Series: A discrete set of label name / value pairs containing one or more timestamped data points Metric: A group of series sharing a “__name__” label value, eg: “api_requests” Cardinality Explosion / High Card. Event: Sharp increase in series creation rate Exploding Label: A label whose count of distinct values is disproportionately high compared to other labels within a metric’s series

slide-7
SLIDE 7

@strofcon

So… Explosions?

Rapid inflation of the number of series under one or more metrics Examples of Causes

  • Prolonged extreme pod turnover rates
  • Highly elastic workloads with fine-to-medium grain labels
  • Bad code deploy that sticks unique IDs, timestamps, or the like into a label

value

○ This one seems to be the most common cause ○ Magnitude tends to be huge

https://openclipart.org/detail/298678/bomb-2

slide-8
SLIDE 8

@strofcon

Why do I care?

Areas of concern: 1. Meaningfulness of affected data

a. Single “legitimate” data point per series, inability to aggregate on “exploding” labels

2. Stability and responsiveness of Prometheus proper

a. Query times, memory usage, scrape durations, remote_write queue, etc.

3. Stability of downstream receiving services

a. Cortex (remote write); BigTable, DynamoDB, Thanos (chunk stores); etc.

https://openclipart.org/detail/196149/fireball https://upload.wikimedia.org/wikipedia/en/3/38/Prometheus_software_logo.svg

slide-9
SLIDE 9

@strofcon

Impact on Prometheus

@strofcon

slide-10
SLIDE 10

@strofcon

Impact on Cortex & BigTable

@strofcon

slide-11
SLIDE 11

@strofcon

Who ya gonna call? Bomb Squad!

Overview: 1. Run as sidecar to Prometheus proper 2. Bootstrap recording rules into Prometheus 3. Monitor for exploding metrics 4. When found, identify exploding label 5. Insert “silencing rule” relabel config(s) …

  • n. CLI commands available to list and unsilence metrics

https://upload.wikimedia.org/wikipedia/en/3/38/Prometheus_software_logo.svg

slide-12
SLIDE 12

@strofcon

In which Cody attempts a live demo...

slide-13
SLIDE 13

@strofcon

cody@freshtracks.io @strofcon

Thanks

github.com/Fresh-Tracks/bomb-squad