Managing Prometheus in a Security-focused Environment Linux - - PowerPoint PPT Presentation

▶
managing prometheus in a security focused environment
SMART_READER_LITE
LIVE PREVIEW

Managing Prometheus in a Security-focused Environment Linux - - PowerPoint PPT Presentation

Sep 08, 2023 340 likes •605 views

Managing Prometheus in a Security-focused Environment Linux Monitoring at HUK-COBURG PromCon 2019 | Christian Hoffmann Conways Law? 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 2 Introduction $ cat

slide-1
SLIDE 1

Managing Prometheus in a Security-focused Environment

Linux Monitoring at HUK-COBURG PromCon 2019 | Christian Hoffmann

slide-2
SLIDE 2

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 2

Conway‘s Law?

slide-3
SLIDE 3

$ cat /HUK-COBURG

Introduction

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 3

HUK-COBURG  German consumer insurance company  Largest car insurance for consumers in Germany  12 million customers  10.000 employees

slide-4
SLIDE 4

$ cat /HUK-COBURG/IT

Introduction

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 4

HUK-COBURG IT-related departments …  800 people  Not a startup, but lots of teams  Highly regulated

slide-5
SLIDE 5

$ cat /HUK-COBURG/IT/Linux

Introduction

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 5

HUK-COBURG Linux Platform Development IT-related departments …  Internal IaaS provider  900 RHEL servers  Two main data centers

slide-6
SLIDE 6

$ cat /HUK-COBURG/IT/Linux/Christian Hoffmann

Introduction

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 6

HUK-COBURG Linux Platform Development IT-related departments …  One of ten people  Joined in 2016  Linux & Open Source enthusiast

slide-7
SLIDE 7

$ cat /HUK-COBURG/IT/Linux/Application owners

Introduction

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 7

HUK-COBURG Linux Platform Development IT-related departments … Application owners  About 130 people, running:

  • Databases
  • Web servers
  • …
slide-8
SLIDE 8

$ cat /HUK-COBURG/IT/Linux/Others

Introduction

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 8

HUK-COBURG Linux Platform Development IT-related departments … Application owners … Operations

slide-9
SLIDE 9

Scraping Alerts Graphs Integrations

Overview

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 9

Monitoring

slide-10
SLIDE 10

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 10

 Close to the target  What does close mean?

Placement of Prometheus Instances

Scraping Firewalled zone #40

SMTP? Alertmanager/HTTP?

slide-11
SLIDE 11

2

Our setup: One Prometheus per DC

Scraping

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 11

DC 1 DC 2 VMs

20 GiB

RAM

1.2 TiB

disk

1.7 M

series

30 k

samples/s

60s

scrape_interval

2 600

file_sd

1 600

rec rules

200

alert rules

4

cores

slide-12
SLIDE 12

# ps –ef | grep agent root 3474 Nov07 00:30:14 /opt/security-scanner/agent root 7182 Nov07 00:05:03 /opt/hardware-monitoring/agent root 1139 Nov07 83:01:37 /opt/license-management/agent root 4100 Nov07 00:20:00 /opt/config-management/agent root 9983 Nov07 01:30:53 /opt/backup-management/agent ...

Scraping: Securing and unifying metrics access

Scraping

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 12

slide-13
SLIDE 13

# nmap server1001 PORT STATE SERVICE 22/tcp open ssh Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds

So…

Scraping

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 13

slide-14
SLIDE 14

Prometheus server

Introducing sshified

Scraping

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 14

Monitoring target

sshified sshd node_exporter

127.0.0.1:9100 proxy_url: 127.0.0.1:8000 10.1.2.3:22

slide-15
SLIDE 15

Exporters

Scraping

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 15

node

textfile runs as root systemd

process

non-systemd procs

multilog blackbox

decentralized

slide-16
SLIDE 16

Scraping Alerts Graphs Integrations

Overview

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 16

Monitoring

slide-17
SLIDE 17

Alertmanager

Alerts

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 17

Operations Alertmanager

  • Routing
  • LinuxPlatform.+
  • LinuxServer.+
  • App.+
  • Integrations
  • webhook
  • email
  • Silences

Server inventory Central Event Management

  • Dead man’s switch
  • Incident creation
  • Paging

syslog

Prometheus

slide-18
SLIDE 18

Scraping Alerts Graphs Integrations

Overview

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 18

Monitoring

slide-19
SLIDE 19

Grafana with basic multi-tenancy

Graphs

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 19

Apache httpd

  • mod_ldap
  • mod_auth_kerb

Template

  • wner_john
  • wner_lisa
  • wner_*

prometheus-filter-proxy

John

127.0.0.1:8888/owner="john"/api/v1/query?query=up 127.0.0.1:9090/api/v1/query?query=up{owner="john"}

Prometheus

huk-grafana-provisioning.py

slide-20
SLIDE 20

Grafana with high availability

Graphs

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 20

Apache httpd prometheus-filter-proxy Prometheus

John 10.1.2.3

Apache httpd prometheus-filter-proxy Prometheus

rsync grafana.sqlite

slide-21
SLIDE 21

Scraping Alerts Graphs Integrations

Overview

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 21

Monitoring

slide-22
SLIDE 22

hiera

common.yml

role/web.yml

role/db.yml

node/srv1001.yml

Integrating Prometheus into Configuration Management

Integrations

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 22

Deploy & configure exporters Role-specific alerts

  • Scrape configs
  • Platform alerts
slide-23
SLIDE 23

 Staging of new Linux patches  Roll-out on application servers

Integrating Patch Management into Prometheus

Integrations

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 23

Development Staging Production

slide-24
SLIDE 24

 Long Term Storage, Downsampling, „Janitor“  Dashboard performance  Lots of additional ideas and areas for work

What‘s up next?

Future

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 24

slide-25
SLIDE 25

Prometheus and Grafana provide us  Sufficient flexibility in a regulated environment,  Basic multi-tenancy for our teams, and  Helpful integrations into other processes.

Benefits & Takeaways

Summary

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 25

slide-26
SLIDE 26

11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment

Thanks! Any questions?

Christian Hoffmann Linux System Engineer at HUK-COBURG christian.hoffmann2@huk-coburg.de

http://github.com/hoffie/sshified http://github.com/hoffie/prometheus-filter-proxy http://github.com/hoffie/multilog_exporter