Alert acknowledgement With Alertmanager ukasz Mierzwa Alert states - - PowerPoint PPT Presentation

▶

Nov 04, 2022 612 likes •754 views

Alert acknowledgement With Alertmanager ukasz Mierzwa Alert states in Alertmanager An alert can be: Unprocessed Active Suppressed (silence or inhibition rule) How to acknowledge an alert with only Alertmanager? Use a silence

SLIDE 1

Alert acknowledgement

With Alertmanager Łukasz Mierzwa

SLIDE 2

Alert states in Alertmanager

An alert can be:

Unprocessed
Active
Suppressed (silence or inhibition rule)

SLIDE 3

SLIDE 4

How to acknowledge an alert with only Alertmanager?

Use a silence

SLIDE 5

Option #1

Create a silence with long duration
Expire it manually when alert stops

SLIDE 6

Option #2

Create a silence with short duration
Keep extending duration as long as

alerts keep firing

SLIDE 7

https://github.com/prymitive/kthxbye

SLIDE 8

$ ./kthxbye 2bd8ab23-eec7-49f4-814f-f8882acf709e is not used by any alert, letting it expire 63599cb9-9766-4cf2-9fcd-035c43312005 expires in 2m34.8882584s and matches 8 alert(s), extending it by 15m0s Silence updated: 63599cb9-9766-4cf2-9fcd-035c43312005 => 81faa1ae-3c42-4619-8b35-b3bc5c409b3a 2bd8ab23-eec7-49f4-814f-f8882acf709e is not used by any alert, letting it expire

SLIDE 9

https://github.com/prymitive/karma

SLIDE 10

SLIDE 11

Live demo

https://karma-demo.herokuapp.com/

SLIDE 12

Takeaways

Not everything needs to be a core Prometheus /

Alertmanager feature

It’s easy to extend Prometheus / Alertmanager
It’s easy to write Alertmanager v2 API clients