Secure and version controlled configuration with Puppet, Hiera and - - PowerPoint PPT Presentation

Secure and version controlled configuration with Puppet, Hiera and GPG

Jens Bräuer <braeuer.jens@gmail.com>

Agenda

• 1. Background
• 2. Puppet for infrastructure automation

2.1 Externalize configuration information 2.2 Problems with externalized information

• 3. Approach to be secure and version

controlled

Puppet ahead - Ask questions please.

Background

Commercetools GmbH (CT)

• full service eCommerce solutions

(Redbull, Brita, ...)

• backed by RackSpace
• total ~40 people

Products

• "current" GRID

○ Play + Tomcat + MySQL + Solr

• "new" Sphere

○ Play + Scala + MongoDB + ElasticSearch

Background

• Operations

○ ~ 60 servers in total (+ some legacy) ■ ~ 20 GRID ■ ~ 20 Sphere ■ ~ 20 "support" (internal, build machines, etc)

• one OS (Ubuntu 12.04, almost, see legacy)
• heavy use of Puppet

○ for development (Vagrant Box) ○ for CI ○ for Staging + Production

Real world example - Configure ElasticSearch

class elasticsearch::server { $elasticsearch_dev_mode = true $elasticsearch_cluster_name = "elasticsearch" $elasticsearch_cluster_members = ["localhost"] package { "elasticsearch": ensure => installed } # uses cluster_name and cluster_members file { "/etc/elasticsearch/elasticsearch.yml":

• wner => root, group => root, mode => 0644,

content => template("elasticsearch/elasticsearch.yml.erb"), } service { "elasticsearch": ensure => running, enable => true, require => [Package["elasticsearch"]] } }

ElasticSearch + Puppet

Simple approach: Hard coded

Hiera to the rescue

extlookup, external node classifier (enc) Hiera, short for “hierarchy” [...] is a pluggable, hierarchical database that can query [...] backends for configuration data.

class elasticsearch::server( $elasticsearch_dev_mode = hiera("elasticsearch.dev.mode") $elasticsearch_cluster_name = hiera("elasticsearch.cluster.name") $elasticsearch_cluster_members = hiera("elasticsearch.cluster.members")) {

Configuration information - by src

elasticsearch.cluster.members:

• search1.sphere.cloud.commercetools.de
• search2.sphere.cloud.commercetools.de

postfix.relay.username: user postfix.relay.password: XXXXXXXXXXX postfix.relay.host: smtp.mailjet.com elasticsearch.dev.mode: true postfix.relay.username: automation@ct.de postfix.relay.password: XXXXXXXXXXXX postfix.relay.host: mail.google.com net.trust: - 1.2.3.4

• 2.3.4.4

sphere-stage sphere global

Configuration information - merged

elasticsearch.dev.mode: true elasticsearch.cluster.members:

• search1.sphere.cloud.commercetools.de
• search2.sphere.cloud.commercetools.de

postfix.relay.username: user postfix.relay.password: XXXXXXXXXXX postfix.relay.host: smtp.mailjet.com net.trust: - 1.2.3.4

• 2.3.4.4

hiera

Hiera to the rescue

Supply default values in-code

$elasticsearch_cluster_name = hiera("elasticsearch.cluster.name", "mycluster")

Overwrite keys in-code

class { "elasticsearch::server": elasticsearch_cluster_name => "othername" }

Hiera to the rescue

Backends to get data from

• YAML
• JSON
• MySQL
• Mongo
• Redis
• ...

Included in Puppet 3.x, Add-on in Puppet 2.7

Share (and contain) config information

Configuration hierarchy

Common GRID Stage Prod Sphere Build Vagrant Stage Prod CI

Needed By Devs Please NO Devs :)

Share config information

Requirements

• version control: plain files
• JSON or YAML
• have sensitive and non-sensitive fields side

by side (hiera-gpg)

• limit visibility of items

Decisions

• separate config in extra repository?
• (only) credentials are sensitive

Share config information

Edited like

mail.relay.enable: true mail.relay.server: in.mailjet.com mail.relay.port: 587 mail.relay.user: aee97cb3ad288ef0add6c6b5b5fae48a mail.relay.password: PLAIN(supersecretpassword)

Stored like

mail.relay.enable: true mail.relay.server: in.mailjet.com mail.relay.port: 587 mail.relay.user: aee97cb3ad288ef0add6c6b5b5fae48a mail.relay.password: ENC(jA0EAwMC96....Wghh/AtP)

Config information - visibility

Common Sphere GRID Stage Prod CI

Potential different Ops-Team root@prod.sphere.ct.de jens@ct.de hajo@ct.de sven@ct.de root@prod.sphere.ct.de root@stage.sphere.ct.de root@ci.sphere.ct.de jens@ct.de hajo@ct.de sven@ct.de

Raziel

http://en.wikipedia.org/wiki/Raziel

commit f8bb8219d166201141df8454aeaa8af0b8009d04 Author: Jens Braeuer <jens.braeuer@commercetools. de> Date: Sat Jan 26 13:54:22 2013 +0000 Add read-only Icinga access. Introduce "icinga.guest.password" key.

Raziel - Keeper of secrets

• Change code & configuration together.
• Secure storage of production configuration

information.

Raziel - Keeper of secrets

• written in Ruby (1.9.x)
• two parts

○ hiera backend to transparently integrate with Puppet ○ command line tool to enable editing PLAIN(...) to ENC(....) to PLAIN(...)

• Encryption

○ based on asymmetric & symmetric encryption ○ GPG + GPG-ME ○ ruby-gpgme

Raziel - Caveats

• comments lost due to Ruby parser
• "re-keying" workflow needs manual steps

Raziel - Some stats

• small

○ <1000 LOC ○ 410 LOC in tests

• Debian package
• Tested on

○ OS-X 10.7 / 10.8 ○ Ubuntu 12.04

• Tested with

○ Puppet 2.7 and 3.0

Will be open-sourced.

Questions?

How do you do this?

Thanks!

Contact me braeuer.jens@gmail.com github.com/jbraeuer Thanks Hajo! Hajo Eichler <eichler.hajo@gmail.com>