Managing Prometheus in a Security-focused Environment Linux Monitoring at HUK-COBURG PromCon 2019 | Christian Hoffmann
Conway‘s Law? 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 2
Introduction $ cat /HUK-COBURG HUK-COBURG German consumer insurance company Largest car insurance for consumers in Germany 12 million customers 10.000 employees 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 3
Introduction $ cat /HUK-COBURG/IT IT-related departments HUK-COBURG 800 people Not a startup, but lots of teams … Highly regulated 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 4
Introduction $ cat /HUK-COBURG/IT/Linux Linux Platform Development IT-related departments HUK-COBURG … Internal IaaS provider 900 RHEL servers Two main data centers 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 5
Introduction $ cat /HUK-COBURG/IT/Linux/Christian Hoffmann Linux Platform Development IT-related departments One of ten people HUK-COBURG Joined in 2016 … Linux & Open Source enthusiast 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 6
Introduction $ cat /HUK-COBURG/IT/Linux/Application owners Linux Platform Development IT-related departments Application owners HUK-COBURG … About 130 people, running: • Databases • Web servers • … 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 7
Introduction $ cat /HUK-COBURG/IT/Linux/Others Linux Platform Development IT-related departments Application owners HUK-COBURG Operations … … 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 8
Overview Scraping Alerts Monitoring Graphs Integrations 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 9
Scraping Placement of Prometheus Instances Close to the target What does close mean? Firewalled zone #40 SMTP? Alertmanager/HTTP? 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 10
Scraping Our setup: One Prometheus per DC DC 1 DC 2 60s 2 4 1.7 M 200 VMs cores series alert rules scrape_interval 20 GiB 1.2 TiB 2 600 1 600 30 k RAM disk file_sd rec rules samples/s 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 11
Scraping Scraping: Securing and unifying metrics access # ps – ef | grep agent root 3474 Nov07 00:30:14 /opt/security-scanner/agent root 7182 Nov07 00:05:03 /opt/hardware-monitoring/agent root 1139 Nov07 83:01:37 /opt/license-management/agent root 4100 Nov07 00:20:00 /opt/config-management/agent root 9983 Nov07 01:30:53 /opt/backup-management/agent ... 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 12
Scraping So… # nmap server1001 PORT STATE SERVICE 22/tcp open ssh Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 13
Scraping Introducing sshified Monitoring target Prometheus server proxy_url: 10.1.2.3:22 127.0.0.1:8000 127.0.0.1:9100 node_exporter sshified sshd 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 14
Scraping Exporters node process multilog blackbox non-systemd textfile decentralized procs runs as root systemd 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 15
Overview Scraping Alerts Monitoring Graphs Integrations 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 16
Alerts Alertmanager Alertmanager Central Event • Routing Management • LinuxPlatform.+ Operations Prometheus • LinuxServer.+ • Dead man’s switch • App.+ • Incident creation • Integrations • Paging • webhook • email • Silences syslog Server inventory 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 17
Overview Scraping Alerts Monitoring Graphs Integrations 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 18
Graphs Grafana with basic multi-tenancy huk-grafana-provisioning.py Template 127.0.0.1:8888/owner="john"/ api/v1/query?query=up Apache httpd owner_john prometheus-filter-proxy • mod_ldap • mod_auth_kerb owner_lisa 127.0.0.1:9090/api/v1/query?query=up{ owner="john" } John owner_* Prometheus 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 19
Graphs Grafana with high availability Apache httpd prometheus-filter-proxy 10.1.2.3 Prometheus rsync grafana.sqlite John Prometheus Apache httpd prometheus-filter-proxy 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 20
Overview Scraping Alerts Monitoring Graphs Integrations 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 21
Integrations Integrating Prometheus into Configuration Management Deploy & configure exporters hiera common.yml role/web.yml • Scrape configs role/db.yml • Platform alerts node/srv1001.yml Role-specific alerts 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 22
Integrations Integrating Patch Management into Prometheus Staging of new Linux patches Development Roll-out on application servers Staging Production 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 23
Future What‘s up next? Long Term Storage, Downsampling, „ Janitor “ Dashboard performance Lots of additional ideas and areas for work 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 24
Summary Benefits & Takeaways Prometheus and Grafana provide us Sufficient flexibility in a regulated environment, Basic multi-tenancy for our teams, and Helpful integrations into other processes. 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment 25
Thanks! Any questions? Christian Hoffmann Linux System Engineer at HUK-COBURG christian.hoffmann2@huk-coburg.de http://github.com/hoffie/sshified http://github.com/hoffie/prometheus-filter-proxy http://github.com/hoffie/multilog_exporter 11/2019 Christian Hoffmann | Managing Prometheus in a Security-focused Environment
Recommend
More recommend
