Managing Configuration Drift and Auditing with Salt Duncan - PowerPoint PPT Presentation

Managing Configuration Drift and Auditing with Salt Duncan Mac-Vicar P. Director, Data Center Management R&D, SUSE dmacvicar@suse.com

How to manage infrastructure? 2

Sysadmin Alexis ● Manages his servers with bash #!/bin/bash scripts. cat <<EOF | ● They reside in`~/bin` server1 ● Strict ownership and approval server2 process. EOF while read line ssh -q user1@${line} zypper up done

New colleague: Devops Adrian /etc/motd: file.managed: - source: salt://common/motd apache: pkg.installed ➔ Writes "Configuration Management" recipes and templates ➔ They reside in `git`.

The two brains of IT Mode 1 Mode 2 Reliability Agility Waterfall, ITIL Agile, DevOps Conventional Projects New & Uncertain Projects Long-cycle Times Short Cycle (days, weeks) (months) 5

Devops Adrian explains “If somebody changes the configuration, I just re-apply it and the tool brings it to the correct state.”

Sysadmin Alexis reads: Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life. 7

Infrastructure as code ➢ Has become a pragmatic way to implement configuration management for IT infrastructure. ➢ We know how to manage change of source code. ➢ We have the tools and processes for it.

Salt 101 Ports: 4505-4506 ØMQ Minion Master Minion Minion

Salt 101 master $ salt ‘*’ pkg.install foo $ salt host1 docker.pause c001 $ salt ‘web*’ cmd.run \ ‘cat /etc/fstab’ configuration results commands $ salt ‘*’ state.apply minions

What is a state? master /etc/motd: file.managed: - source: salt://common/motd apache: pkg.installed configuration results commands minions

States “state” is how Salt calls configuration, in its declarative form.

Non-Compliant system $ salt minion1 state.apply test=True minion1: ---------- ID: /etc/motd Function: file.managed Result: None Comment: The file /etc/motd is set to be changed Started: 10:06:05.021643 Duration: 30.339 ms Changes: ---------- diff: --- +++ @@ -1 +1 @@ -Have a lot of fun... +This is my managed motd Summary for minion1 ------------ Succeeded: 1 (unchanged=1, changed=1) Failed: 0 ------------ Total states run: 1

New trainee $ useradd -r mudserver

Let’s run it again $ salt minion1 state.apply test=True minion1: ---------- ID: /etc/motd Function: file.managed Result: None Comment: The file /etc/motd is set to be changed Started: 10:06:05.021643 Duration: 30.339 ms Changes: ---------- diff: --- +++ @@ -1 +1 @@ -Have a lot of fun... +This is my managed motd Summary for minion1 ------------ Succeeded: 1 (unchanged=1, changed=1) Failed: 0 ------------ Total states run: 1

The change was not detected It was not part of the # We can express... configuration. joe: user.present # How to express? any other: user.absent

Disappointed Sysadmin ● Devops tools focus in creating new systems. ● Not all change accounted. “Is it really Configuration Management”. ● What they call "Configuration Management" is really "Automation". ● The novelty is more about the declarative approach (vs imperative).

Are “Classic” IT priorities the same? ● Detecting Configuration Drift. ● Auditing Compliance. Drift ● Documenting infrastructure. Audit Document

“Incomplete” Configuration States Templates Baseline

Where is the baseline? In configuration management, a "baseline" is an agreed description of the attributes of a product, at a point in time, which serves as a basis for defining change. ~~MIL-HDBK-61

How to define a baseline? How to integrate it with the rest of the configuration?

Snapper (http://snapper.io) ➔ snapper is to snapshots what zypper/apt-get/dnf are to packages. ➔ First demoed in SUSECon 2011 ! ➔ Main feature of SUSE Linux Enterprise 12

Created by SUSE, available everywhere ● https://wiki.archlinux.org/index.php/Snapper ● https://apps.fedoraproject.org/packages/snapper ● https://packages.debian.org/search?keywords=snapper (don't forget to mention btrfs)

Snapper 101 snapper list-configs snapper list snapper create snapper mount <number> snapper status <number1>..<number2> snapper diff <number1>..<number2> [files] snapper undochange <number1>..<number2> [files]  YaST and zypper take snapshots automatically  In grub menu you can boot old snapshots

YaST2 snapper

Salt and Snapper integration I salt '*' snapper.list_snapshots master salt '*' snapper.get_snapshot salt '*' snapper.create_snapshot salt '*' snapper.undo salt '*' snapper.diff configuration results commands minions

Salt andSnapper Integration $ salt minion2 snapper.run function=file.append args='["/etc/motd", "some text"]' minion2: Wrote 1 lines to "/etc/motd" ... pre | 21 | | Thu.. | root | number | salt job 6668 | salt_jid=6668 post | 22 | 21 | Thu... | root | number | salt job 6668 | salt_jid=6668

Salt and Snapper integration $ salt minion2 snapper.diff_jid 6668 minion2: ---------- /etc/motd: --- /.snapshots/21/snapshot/etc/motd +++ /.snapshots/22/snapshot/etc/motd @@ -1 +1,2 @@ Have a lot of fun... +some text snapper.undo_jid also works

State module Back to the baseline problem, imagine you could say: # Starting from snapshot #3 baseline: States Templates - ???? # then ... /etc/motd: Baseline file.managed: - source: salt://common/motd apache: pkg.installed

State module You can! my_baseline: snapper.baseline_snapshot: States Templates - number: 20 - ignore: - /var/log Baseline - /var/cache /etc/motd: file.managed: - source: salt://common/motd

If the somebody adds a new user, a drift against the baseline rule will happen: $ salt minion1 state.apply test=True minion1: ---------- ID: my_baseline Function: snapper.baseline_snapshot Result: None Comment: 1 files changes are set to be undone ... Changes: ... /etc/passwd: ... diff: --- /etc/passwd +++ /.snapshots/21/snapshot/etc/passwd @@ -22,5 +22,3 @@ duncan:x:1000:100:Duncan Mac-Vicar P.:/home/duncan:/bin/zsh -mudserver:x:167:100::/home/mudserver:/bin/bash ---------- ID: /etc/motd ... Succeeded: 2 (unchanged=2, changed=2)

Applying states If you apply the state (eg. no `test=True`), the system will be set to the state of the baseline snapshot before applying the rest of the states. $ salt minion1 state.apply Current Baseline state ... states

Managing snapshots by number? Creates a snapshot and adds a $ salt '*' snapper.create_baseline “baseline” tag to the “userdata” property of each snapshot. Type # Pre Date Desc userdata single 0 current ... post 20 19 Sept 26... important=no ... single 22 Sept 26... baseline_tag=baseline

Baseline tags ● You can move the baseline, last_production: without affecting your state. snapper.baseline_snapshot: - tag: baseline ● The last tagged snapshot will be used. e n i C l e o s n a f i B g u r e Audit t n e m D u c r i o f t D

Salt Snapper module ● Already submitted upstream. Will be part of Carbon release. ● Also available in SUSE Linux Enterprise/SUSE Manager 2015.8.x Salt package ● Carbon also supports automatic snapshots when applying states https://docs.saltstack.com/en/develop/topics/releases/carbon.html

(about state snapshots) $ salt minion2 snapper.run function=state.apply

Other Resources to Manage Drift

Salt Survey Runner Module Survey groups the returned values in pools of unique results. salt-run survey.diff survey_sort=up "*" cmd.run 'cat /etc/hosts' This tells you which server differs from the others. v2 /etc/hosts v1 /etc/hosts

Salt Package Module salt 'web*' pkg.diff /etc/sudoers Tells the difference between the `/etc/sudoers` of the original package vs the installed one.

Hubble (http://hubblestack.io) Tool Purpose Nova Auditing Framework Pulsar File integrity monitor, security events Nebula Query infrastructure security snapshots Quasar Reporting

Available Nova modules ● grep (configuration values) ● iptables (firewall rules) ● netstat (listening ports) ● openscap (CVE scan) ● openssl (cert validation & expiration) ● pkg (installed packages) ● service (running services) ● stat (ownerships & permissions) ● sysctl (kernel parameters) ● vulners.com (CVE scan)

Future work

The two brains of IT Mode 1 Mode 2 Reliability Agility Waterfall, ITIL Agile, DevOps Conventional Projects New & Uncertain Projects Long-cycle Times Short Cycle (days, weeks) (months) 44

“Bimodal Datacenter” Softwar Mode 2 Mode 1 e Defined Magnum *: Comput Containers e Storage Network High Deployment Networking Scaling Monitoring Availability 45

Docker images 46