Use Saltstack to deploy a full monitoring and supervision stack - - PowerPoint PPT Presentation

▶

Nov 12, 2022 390 likes •667 views

Use Saltstack to deploy a full monitoring and supervision stack #cfgmgmtcamp18 Arthur Lutz Logilab #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab (part of) who am I Arthur Lutz Logilab Python dev / Sysadmin / Debian /

SLIDE 1

Use Saltstack to deploy a full monitoring and supervision stack

#cfgmgmtcamp18 Arthur Lutz ‐ Logilab

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 2

(part of) who am I

Arthur Lutz Logilab Python dev / Sysadmin / Debian / Devops SaltStack Certied Engineer Paris Salt Meetup Python Nantes Meetup / Nantes Monitoring Meetup

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 3

Big picture ‐ scienfic cloud applicaon

Complex scaling parameters salt-based auto-scaling, scale up and down based

n application logic

scaling is based on metrics collected by salt automatically deploy application-oriented monitoring

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 4

dashboards from diverse data

grafana provides data source diversity we're looking for data origin diversity, and agile ways to deploy them system (internal) data cloud (external) data application-specic data autoscaling state, etc...

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 5

agility by design consistency by design

easily dene and add metrics compare and test metrics from different points of view

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 6

Overview of components

SaltStack - huge toolkit to manage infrastructure Netdata - collect metrics, visualise "realtime" status of server Sensu - devops oriented monitoring system Graphite / Carbon - to collect and query time series data Grafana - to build and display dashboards on the collected data

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 7

Components used

salt pillars salt scheduler salt mine salt returners salt runners salt reactor salt-cloud ... "which ones did we not use?"

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 8

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 9

Salt formulas

salt-formula (what else?) nginx-formula nfs-formula nagios-formula sensu-formula rabbitmq-formula redis-formula

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 10

Role based approach

Pillars (with pillar_merge_list: True ) :

base: '*':

role.sensu-client

'saltmaster.local':

role.sensu-server

States :

base: 'I@role:sensu-client':

sensu.client

'I@role:sensu-server':

sensu.server

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 11

Salt mine

send unique ID used by Netdata to stream clients to central instance states :

{{ salt['mine.get']('role:netdata-master', 'network.interface_ip' expr_form='pillar').values()[0] }}

use to retrieve location of Rabbitmq server used by

Sensu . pillars :

{% set rabbitmq = salt.saltutil.runner("mine.get", tgt='role:rabbitmq', fun="network.interface_ip", tgt_type='pillar').values() %}

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 12

Salt returners

Carbon returner to push metrics from application

and infrastructure returner address is congured in pillars (cf extraction from mine)

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 13

Salt custom modules

Custom modules are shipped by salt master from

_modules folder

celery statistics (use app.inspect.stats ) core application metrics using HTTP API sentry (error collector) metrics using the sentry python binding (which uses the sentry HTTP API)

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 14

Custom module ‐ celery inspect

def stats(): ''' Provide stats from celery tasks via app.inspect.stats() [snip] ''' ret = __salt__['redis.sentinel_get_master_ip']('name') r_url = 'redis://{}:{}/0' app = Celery(broker=r_url.format(ret['master_host'], ret['master_port'])) stats = app.control.inspect().stats() _clean(stats) return stats

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 15

Salt‐cloud

enables custom auto-scaling system portable between cloud infrastructures enables to run infrastructure on local openstack (or even devstack)

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 16

Salt reactor ‐ event driven!

react to new VM connecting get info about VM (for example kernelversion ) upgrade kernel to linux-backports reboot system deploy monitoring deploy application Ended up being replaced by Packer generated AMI.

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 17

Let's zoom in

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 18

Visualising complexity

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 19

monitoring ‐ how

launch a salt runner to pool the metrics

salt.execute runner is used to get infrastructure

view of a given metric

cloud.profile runner is used to provision new VMs cloud.destroy runner is used to scale down unused

VMs

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 20

monitoring ‐ how

Netdata sends system metrics

central Netdata sends data to Graphite

Salt sends custom metrics to Graphite Sensu executes checks (subscriber based model) Sensu collects metrics and sends them to Graphite Grafana uses the metrics stored in Graphite to

display dashboards

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 21

MASTER

sensuserver sensuapi

web

minion

storage

minion

client

compute node

minion

client

compute node

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 22

Demo me!

install netdata role based architecture auto congure streaming by pushing the API id to the mine install graphite deploy netdata graphite conguration install grafana congure grafana datasource with salt congure grafana dashboard with salt

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 23

Demo me!

install rabbitmq with rabbitmq-formula for sensu install sensu with sensu-formula use roles to congure subscribed sensu checks congure uchiwa for web ui congure sensu metrics to send data to graphite update grafana dashboard with salt

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 24

What's next / Roadmap

convert schedulers to beacons various optimisations use reactor instead of polling look into salt's thorium (complex reactor system)

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 25

End

Thanks for your attention Questions ? Comments ? Improvements ?

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

SLIDE 26

sources

http://slides.logilab.fr/2018/cfgmgmtcamp_saltsta ck_monitoring.pdf Demo : https://github.com/arthurlogilab/salt- vagrant-demo Contact / Follow us : https://www.logilab.fr @logilab @arthurlutz arthurlutz@social.logilab.org

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab