Managing Configuration Drift and Auditing with Salt Duncan - - PowerPoint PPT Presentation

Managing Configuration Drift and Auditing with Salt

Duncan Mac-Vicar P. Director, Data Center Management R&D, SUSE dmacvicar@suse.com

2

How to manage infrastructure?

#!/bin/bash cat <<EOF | server1 server2 EOF while read line ssh -q user1@${line} zypper up done

Sysadmin Alexis

• Manages his servers with bash

scripts.

• They reside in`~/bin`
• Strict ownership and approval

process.

New colleague: Devops Adrian

➔Writes "Configuration

Management" recipes and templates

➔They reside in `git`.

/etc/motd: file.managed:

• source: salt://common/motd

apache: pkg.installed

5

The two brains of IT

Mode 2 Mode 1

Agile, DevOps Waterfall, ITIL New & Uncertain Projects Conventional Projects Short Cycle (days, weeks) Long-cycle Times (months) Agility Reliability

Devops Adrian explains

“If somebody changes the configuration, I just re-apply it and the tool brings it to the correct state.”

7

Sysadmin Alexis reads:

Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life.

Infrastructure as code

➢Has become a pragmatic way to implement configuration

management for IT infrastructure.

➢We know how to manage change of source code. ➢We have the tools and processes for it.

Salt 101

Ports: 4505-4506 Master Minion Minion Minion ØMQ

Salt 101

configuration commands results master minions

$ salt ‘*’ pkg.install foo $ salt host1 docker.pause c001 $ salt ‘web*’ cmd.run \ ‘cat /etc/fstab’ $ salt ‘*’ state.apply

What is a state?

configuration commands results master minions

/etc/motd: file.managed:

• source: salt://common/motd

apache: pkg.installed

“state” is how Salt calls configuration, in its declarative form.

States

Non-Compliant system

$ salt minion1 state.apply test=True

minion1:

• ID: /etc/motd

Function: file.managed Result: None Comment: The file /etc/motd is set to be changed Started: 10:06:05.021643 Duration: 30.339 ms Changes:

• diff:
• +++

@@ -1 +1 @@

• Have a lot of fun...

+This is my managed motd Summary for minion1

• Succeeded: 1 (unchanged=1, changed=1)

Failed: 0

• Total states run: 1

New trainee

$ useradd -r mudserver

Let’s run it again

$ salt minion1 state.apply test=True

minion1:

• ID: /etc/motd

Function: file.managed Result: None Comment: The file /etc/motd is set to be changed Started: 10:06:05.021643 Duration: 30.339 ms Changes:

• diff:
• +++

@@ -1 +1 @@

• Have a lot of fun...

+This is my managed motd Summary for minion1

• Succeeded: 1 (unchanged=1, changed=1)

Failed: 0

• Total states run: 1

The change was not detected

It was not part of the configuration. # We can express... joe: user.present # How to express? any other: user.absent

• Devops tools focus in creating new systems.
• Not all change accounted. “Is it really Configuration Management”.
• What they call "Configuration Management" is really "Automation".
• The novelty is more about the declarative approach (vs imperative).

Disappointed Sysadmin

Are “Classic” IT priorities the same?

• Detecting Configuration Drift.
• Auditing Compliance.
• Documenting infrastructure.

Drift Audit

Document

“Incomplete” Configuration

Baseline

States Templates

Where is the baseline?

In configuration management, a "baseline" is an agreed description of the attributes of a product, at a point in time, which serves as a basis for defining change. ~~MIL-HDBK-61

How to define a baseline? How to integrate it with the rest of the configuration?

Snapper (http://snapper.io)

➔snapper is to snapshots what

zypper/apt-get/dnf are to packages.

➔First demoed in SUSECon 2011! ➔Main feature of SUSE Linux

Enterprise 12

Created by SUSE, available everywhere

• https://wiki.archlinux.org/index.php/Snapper
• https://apps.fedoraproject.org/packages/snapper
• https://packages.debian.org/search?keywords=snapper

(don't forget to mention btrfs)

Snapper 101

snapper list-configs snapper list snapper create snapper mount <number> snapper status <number1>..<number2> snapper diff <number1>..<number2> [files] snapper undochange <number1>..<number2> [files]

YaST and zypper take snapshots automatically In grub menu you can boot old snapshots

YaST2 snapper

Salt and Snapper integration I

configuration commands results master minions

salt '*' snapper.list_snapshots salt '*' snapper.get_snapshot salt '*' snapper.create_snapshot salt '*' snapper.undo salt '*' snapper.diff

Salt andSnapper Integration

$ salt minion2 snapper.run function=file.append args='["/etc/motd", "some text"]' minion2: Wrote 1 lines to "/etc/motd" ... pre | 21 | | Thu.. | root | number | salt job 6668 | salt_jid=6668 post | 22 | 21 | Thu... | root | number | salt job 6668 | salt_jid=6668

Salt and Snapper integration

$ salt minion2 snapper.diff_jid 6668 minion2:

• /etc/motd:
• -- /.snapshots/21/snapshot/etc/motd

+++ /.snapshots/22/snapshot/etc/motd @@ -1 +1,2 @@ Have a lot of fun... +some text

snapper.undo_jid also works

State module

Back to the baseline problem, imagine you could say:

# Starting from snapshot #3 baseline:

• ????

# then ... /etc/motd: file.managed:

• source: salt://common/motd

apache: pkg.installed

Baseline

States Templates

State module

You can!

my_baseline: snapper.baseline_snapshot:

• number: 20
• ignore:
• /var/log
• /var/cache

/etc/motd: file.managed:

• source: salt://common/motd

Baseline

States Templates

If the somebody adds a new user, a drift against the baseline rule will happen:

$ salt minion1 state.apply test=True minion1:

• ID: my_baseline

Function: snapper.baseline_snapshot Result: None Comment: 1 files changes are set to be undone ... Changes: ... /etc/passwd: ... diff:

• -- /etc/passwd

+++ /.snapshots/21/snapshot/etc/passwd @@ -22,5 +22,3 @@ duncan:x:1000:100:Duncan Mac-Vicar P.:/home/duncan:/bin/zsh

• mudserver:x:167:100::/home/mudserver:/bin/bash
• ID: /etc/motd

... Succeeded: 2 (unchanged=2, changed=2)

If you apply the state (eg. no `test=True`), the system will be set to the state of the baseline snapshot before applying the rest of the states.

Applying states

$ salt minion1 state.apply

Current

Baseline state

... states

Managing snapshots by number?

$ salt '*' snapper.create_baseline

Creates a snapshot and adds a “baseline” tag to the “userdata” property of each snapshot.

Type # Pre Date Desc userdata single current ... post 20 19 Sept 26... important=no ... single 22 Sept 26... baseline_tag=baseline

Baseline tags

• You can move the baseline,

without affecting your state.

• The last tagged snapshot will be

used.

last_production: snapper.baseline_snapshot:

• tag: baseline

C

• n

f i g u r e D r i f t D

• c

u m e n t B a s e l i n e Audit

Salt Snapper module

• Already submitted upstream. Will be part of Carbon release.
• Also available in SUSE Linux Enterprise/SUSE Manager 2015.8.x Salt

package

• Carbon also supports automatic snapshots when applying states

https://docs.saltstack.com/en/develop/topics/releases/carbon.html

(about state snapshots)

$ salt minion2 snapper.run function=state.apply

Other Resources to Manage Drift

Salt Survey Runner Module

Survey groups the returned values in pools of unique results.

salt-run survey.diff survey_sort=up "*" cmd.run 'cat /etc/hosts' This tells you which server differs from the others.

v1

/etc/hosts

v2

/etc/hosts

Salt Package Module

salt 'web*' pkg.diff /etc/sudoers Tells the difference between the `/etc/sudoers` of the original package vs the installed one.

Hubble (http://hubblestack.io)

Tool Purpose Nova Auditing Framework Pulsar File integrity monitor, security events Nebula Query infrastructure security snapshots Quasar Reporting

Available Nova modules

• grep (configuration values)
• iptables (firewall rules)
• netstat (listening ports)
• openscap (CVE scan)
• openssl (cert validation & expiration)
• pkg (installed packages)
• service (running services)
• stat (ownerships & permissions)
• sysctl (kernel parameters)
• vulners.com (CVE scan)

Future work

44

The two brains of IT

Mode 2 Mode 1

Agile, DevOps Waterfall, ITIL New & Uncertain Projects Conventional Projects Short Cycle (days, weeks) Long-cycle Times (months) Agility Reliability

45

“Bimodal Datacenter”

Deployment High Availability Scaling Monitoring Networking

Softwar e Defined *: Comput e Storage Network Magnum Containers Mode 1 Mode 2

46

Docker images

47

• You bought into the hype and automated everything with Salt.
• You have Salt states and templates for all your infrastructure.

salt://happy

• Now you wand to deploy an app into a container and you need a

container image...

Motivation

48

Dockerfiles

FROM ubuntu:12.04 RUN zypper -n in foo RUN echo "key=val" > /etc/foo/config

49

Building images

foo: pkg.installed /etc/foo/config: file.managed:

• source:

salt://myapp/foo.config salt mybuilder0 dockerng.state myapp mods=myapp

50

Benefits

 Reuse states and templates across workloads.  Access templates, pillar data (eg. secrets) directly from Salt.  Access to Salt modules for configuration.  Implemented as ability to run arbitrary Salt modules inside running

containers.

 Auditing

51

Thanks for listening

53