Configuration management with Chef Edd Dumbill edd@oreilly.com - - PowerPoint PPT Presentation

▶
configuration management with chef
SMART_READER_LITE
LIVE PREVIEW

Configuration management with Chef Edd Dumbill edd@oreilly.com - - PowerPoint PPT Presentation

Apr 16, 2023 307 likes •1.4k views

Configuration management with Chef Edd Dumbill edd@oreilly.com OSCON 2009 Monday, 20 July 2009 About me Created Expectnation, event software that runs OReilly Conferences Co-chair of OSCON Perennial tinkerer and author (most

slide-1
SLIDE 1

Configuration management with Chef

Edd Dumbill edd@oreilly.com OSCON 2009

Monday, 20 July 2009

slide-2
SLIDE 2

About me

  • Created Expectnation, event software that

runs O’Reilly Conferences

  • Co-chair of OSCON
  • Perennial tinkerer and author (most

recently “Learning Rails”)

Monday, 20 July 2009

slide-3
SLIDE 3

Today’s tutorial

  • Overview of Chef
  • Learn by example
  • Common usage patterns
  • Moving on

Monday, 20 July 2009

slide-4
SLIDE 4

Meta

  • Please rate this talk and leave

comments

  • If you’re twittering
  • I’m @edd
  • Hashtag is #oscon
  • Asking questions

Monday, 20 July 2009

slide-5
SLIDE 5

About you

Monday, 20 July 2009

slide-6
SLIDE 6

Overview

Monday, 20 July 2009

slide-7
SLIDE 7

Configuration management

  • Creating and maintaining consistency
  • Installing, updating, reporting
  • Rich history in open source tools
  • cfengine through to Puppet

Monday, 20 July 2009

slide-8
SLIDE 8

Today’s needs

  • Developers are becoming ops people
  • Web architectures and cloud computing
  • Agile sysadmin should complement agile

development

Monday, 20 July 2009

slide-9
SLIDE 9

Developers want

  • Don’t Repeat Yourself
  • Revision control

Monday, 20 July 2009

slide-10
SLIDE 10

Chef

  • Client-server architecture
  • Embraces modern web technologies
  • Written in Ruby

Monday, 20 July 2009

slide-11
SLIDE 11

Chef

  • Cleeent-serfer

ercheetectoore-a

  • Imbreces mudern veb

technulugeees

  • Vreettee in Rooby
  • Bork bork bork

Monday, 20 July 2009

slide-12
SLIDE 12

Chef

  • Has revision control at its core
  • Doesn’t make you learn a new language
  • Comes from a culture of testability and

predictability

Monday, 20 July 2009

slide-13
SLIDE 13

Chef vs Puppet

Monday, 20 July 2009

slide-14
SLIDE 14

Chef vs Puppet

  • Because we needed another open source

war

  • Objective differences
  • Subjective differences
  • Chef has had chance to learn from several

years of Puppet

Monday, 20 July 2009

slide-15
SLIDE 15

Architecture

Chef-client Ohai Node Chef Server Chef-client Ohai Node Chef-client Ohai Node Chef-client Ohai Client Chef Indexer

Monday, 20 July 2009

slide-16
SLIDE 16

Getting started

Monday, 20 July 2009

slide-17
SLIDE 17

Assemble your victims

  • Use VMs for testing environment
  • Ubuntu 8.10 or newer is the sweet spot
  • VirtualBox is a free virtualization tool
  • Identify a server and one or more clients

Monday, 20 July 2009

slide-18
SLIDE 18

Prerequisites

  • Two stage install: basics & bootstrap
  • Minimal prerequisites: Ruby & RubyGems
  • Install via Gems: ohai and chef
  • Bootstrap differs for server and client
  • Packages coming soon

Monday, 20 July 2009

slide-19
SLIDE 19

Server

  • Apache + Passenger
  • Provides administrative Web UI
  • Users identified by OpenID
  • Recipes defined by your chef repository

Monday, 20 July 2009

slide-20
SLIDE 20

Client

  • Invocation of chef-client
  • One-time
  • As a daemon

chef-client -i 3600 -s 600

Monday, 20 July 2009

slide-21
SLIDE 21

Chef repository

  • Contains configuration and cookbooks
  • Clone the Opscode template to start
  • Copy your configuration

Monday, 20 July 2009

slide-22
SLIDE 22

First look at the server

Monday, 20 July 2009

slide-23
SLIDE 23

First client run

Monday, 20 July 2009

slide-24
SLIDE 24

Node attributes

  • Explore with Web UI
  • OS attributes provided by ohai
  • Other attributes are configured by the

installed cookbooks

  • Attributes are mutable

Monday, 20 July 2009

slide-25
SLIDE 25

Making a cookbook

  • Cookbook is the unit of reuse in Chef
  • Unsurprisingly, it contains recipes
  • Generate one with

rake new_cookbook COOKBOOK=hello_world

Monday, 20 July 2009

slide-26
SLIDE 26

Inside the cookbook

  • metadata.rb — cookbook metadata
  • attributes — variables
  • recipes — list of instructions (“resources”)
  • files — files used by resources
  • templates — ERB templates

Monday, 20 July 2009

slide-27
SLIDE 27

Inside the cookbook

  • roles — collections of recipes and attributes
  • definitions — macros of resources
  • libraries — Ruby to extend Chef DSL

Monday, 20 July 2009

slide-28
SLIDE 28

Metadata

  • Functionality similar to metadata in package

management

  • Human readable docs
  • Dependency declarations

Monday, 20 July 2009

slide-29
SLIDE 29

Define an attribute

  • Simple attribute

attributes/my_name.rb

my_name “John Henry”

Monday, 20 July 2009

slide-30
SLIDE 30

A simple recipe

template “/tmp/hello_world.txt” do source “hello_world.txt.erb” variables :my_name => node[:my_name] mode 00664 action :create end

  • recipes/default.rb

Monday, 20 July 2009

slide-31
SLIDE 31

The template

  • templates/default/hello_world.txt.erb

Hello, <%= @my_name %>, how are you today?

Monday, 20 July 2009

slide-32
SLIDE 32

Running the recipe

  • Add the recipe to the node’s recipe list
  • Invoke chef-client
  • Default chef-client setup has client invoked

periodically

Monday, 20 July 2009

slide-33
SLIDE 33

When chef-client runs

  • Node authenticates with server
  • Libraries, attributes, definitions & recipes

are synchronized

  • Libraries, attributes, definitions & recipes

compiled

  • Node state is converged
  • Everything happens on the node

Monday, 20 July 2009

slide-34
SLIDE 34

Attributes & resources

Monday, 20 July 2009

slide-35
SLIDE 35

Attributes

  • May be simply defined, e.g.

my_name “John Henry”

  • Allow overriding, e.g.

my_name “John Henry” unless attribute? (“my_name”)

  • List values are regular arrays

[“foo”, “bar”, “whizz”]

Monday, 20 July 2009

slide-36
SLIDE 36

Attribute hashes

  • Logical groupings of configuration

information, e.g. Apache settings, network interface properties

  • Class used is Mash (from extlib)
  • so you can use :foo or ‘foo’ as a key

Monday, 20 July 2009

slide-37
SLIDE 37

Advanced attributes

  • Methods: attribute?() & recipe?()
  • Access to recipes array

recipes << “hello_world” unless recipe?(“hello_world”)

Monday, 20 July 2009

slide-38
SLIDE 38

Resources

  • The steps that make up a recipe

package “git-core” do action :install end

  • Resources are implemented via Providers

Monday, 20 July 2009

slide-39
SLIDE 39

Package

package "tar" do version "1.16.1-1" action :install end

  • Action can be install, upgrade, remove,

purge

  • Version is optional

Monday, 20 July 2009

slide-40
SLIDE 40

Ruby gems

  • Install gems with package too

package “capistrano” do provider Chef::Provider::Package::Rubygems end

  • Easier:

gem_package “capistrano”

  • Can use source attribute for gem source

Monday, 20 July 2009

slide-41
SLIDE 41

Remote files

  • Copying remote files is easy

remote_file “/tmp/foo.png” do source “foo.png”

  • wner “root”

group “root” mode 0444 action :create end

  • Where does the file live?

Monday, 20 July 2009

slide-42
SLIDE 42

Search path

  • Files and templates are searched for in the

following order: FQDN, platform-version, platform, default

  • For Ubuntu 9.04:

myhost.example.com

ubuntu-9.04 ubuntu default

Monday, 20 July 2009

slide-43
SLIDE 43

More remote file fun

  • File source can be a URL

source “http://warez.com/thing.tgz”

  • Provide SHA256 hash to prevent needless

downloading from chef-server each time

checksum “08da0021”

Monday, 20 July 2009

slide-44
SLIDE 44

Links

  • Symbolic or hard links

link “/usr/bin/randomthing1.8” do to “/usr/bin/randomthing” end

  • Use link_type :hard or :symbolic

(default)

Monday, 20 July 2009

slide-45
SLIDE 45

File

  • Control existence and attributes of a file,

not its contents

file “/tmp/whatever” do

  • wner “root”

group “root” mode “0644” action :create end

  • Other actions are touch, delete

Monday, 20 July 2009

slide-46
SLIDE 46

Other FS resources

  • directory — analog of the File resource
  • remote_directory — recursive remote

copy

Monday, 20 July 2009

slide-47
SLIDE 47

Service

  • Control system services from /etc/init.d and

friends

  • We can en/disable, start, stop & restart

service “my_daemon” do supports :restart => true action [ :enable, :start ] end

Monday, 20 July 2009

slide-48
SLIDE 48

Other resources

  • User
  • Group
  • Cron
  • Route
  • Mount

Monday, 20 July 2009

slide-49
SLIDE 49

Execute

  • Execute arbitrary command

command “mysql-stuff” do execute “/usr/bin/mysql </tmp/ foo.sql” creates “/tmp/outfile.sql” environment {‘FOO’ => “bar”} action :run end

Monday, 20 July 2009

slide-50
SLIDE 50

Script

  • bash, perl, python, ruby, csh

bash “install_foo” do user “root” cwd “/tmp” code <<-EOC wget http://example.org/foo.tgz tar xvf foo.tgz && cd foo ./configure && make install EOC end

Monday, 20 July 2009

slide-51
SLIDE 51

HTTP Request

  • Useful for connecting to existing services

http_request “say_hello” do url “http://myserv.local/check_in” message :node => node[:fqdn] action :post end

  • Posts a JSON payload
  • GET by default

Monday, 20 July 2009

slide-52
SLIDE 52

Resource tricks

Monday, 20 July 2009

slide-53
SLIDE 53

Notifies

  • Chain actions

template “/etc/my_daemon/my.cnf” do source “my.cnf.erb” notifies :restart, resources(:service => “my_daemon”) end

  • By default, notification postponed until end
  • f run, add :immediately as final argument

to override

Monday, 20 July 2009

slide-54
SLIDE 54

Action :nothing

  • If you want a resource to run only on a

notify, specify action :nothing

execute "index-gem-repository" do command "gem generate_index -d /srv/ gems" action :nothing end

Monday, 20 July 2009

slide-55
SLIDE 55

Conditional resources

  • Use only_if and not_if to control resource

execution

  • Takes either shell commands or Ruby

blocks, e.g.

  • nly_if do

IO.read(“/tmp/foo”).chomp == ‘bar’ end

Monday, 20 July 2009

slide-56
SLIDE 56

Platform specifics

  • Selective resource execution
  • nly_if do platform?(“ubuntu”) end
  • Alter package name

package "libwww-perl" do case node[:platform] when "centos" name "perl-libwww-perl" end action :upgrade end

Monday, 20 July 2009

slide-57
SLIDE 57

Roles

Monday, 20 July 2009

slide-58
SLIDE 58

What roles do

  • Bundle recipes and attributes

name "webserver" description "The base role for systems that serve HTTP traffic" recipes "apache2", "apache2::mod_ssl" default_attributes "apache2" => { "listen_ports"=> [ "80", "443" ] }

  • verride_attributes "apache2" =>

{ "max_children"=> "50" }

Monday, 20 July 2009

slide-59
SLIDE 59

What roles are for

  • Convenient way of assigning bundles of

functionality to servers

  • Allow top-level configuration with minimal

need to write new recipes

Monday, 20 July 2009

slide-60
SLIDE 60

Creating roles

  • Ad-hoc from the Web UI
  • As Ruby or JSON from your chef

repository

Monday, 20 July 2009

slide-61
SLIDE 61

Opscode Cookbook

Monday, 20 July 2009

slide-62
SLIDE 62

Opscode cookbooks

  • http://github.com/opscode/cookbooks
  • Integral part of the Chef project
  • If you want it, it’s probably already there
  • common configurations
  • smoothing over platform specifics

Monday, 20 July 2009

slide-63
SLIDE 63

Using the cookbooks

  • Keep your own stuff in site-cookbooks
  • Use git to add cookbooks as a submodule

git submodule add git://github.com/opscode/cookbooks.git cookbooks git submodule init git submodule update

Monday, 20 July 2009

slide-64
SLIDE 64

3rd party cookbooks

  • The cookbook_path from the server config

specifies precedence

  • By default site-cookbooks overrides

cookbooks

  • You can adapt recipes simply by replacing

the parts you wish

Monday, 20 July 2009

slide-65
SLIDE 65

apache2 cookbook

  • Attributes configure basic preferences

(ports, timeout, keepalive)

  • Default recipe sets up sane configuration
  • apache2:: namespace includes recipes for

common modules

Monday, 20 July 2009

slide-66
SLIDE 66

Overriding attributes

  • If you control cookbook, easy enough to

set a default

  • Per-node customizations can be made in

the UI

  • To set new defaults, override selectively in

site-cookbooks

Monday, 20 July 2009

slide-67
SLIDE 67

apache2 definitions

  • Macro for a2ensite & friends

apache_site “my_app” :enable => true end

  • web_app — wraps most of the common

configuration for a web app (e.g. Rails)

Monday, 20 July 2009

slide-68
SLIDE 68

mysql cookbook

  • mysql::client, mysql::server
  • EC2-aware

Monday, 20 July 2009

slide-69
SLIDE 69

Rails cookbook

  • Provides installation recipe and attributes

for tuning

  • rails[:version]
  • rails[:environment]
  • rails[:max_pool_size]
  • Provides web_app template you can copy

Monday, 20 July 2009

slide-70
SLIDE 70

Chef and Rails

Monday, 20 July 2009

slide-71
SLIDE 71

How Chef can help

  • Configuration
  • Deployment
  • Configuration is the better trodden path

Monday, 20 July 2009

slide-72
SLIDE 72

Example configuration

  • Naive Chef recipe to get all the prequisites

in place for an instance of Expectnation

Monday, 20 July 2009

slide-73
SLIDE 73

Worked example

  • Create and deploy a basic Rails app

Monday, 20 July 2009

slide-74
SLIDE 74

chef-deploy

  • A resource that implements Rails

application deployment

  • Models Capistrano’s cached_deploy
  • In rapid development, used at EngineYard
  • http://github.com/ezmobius/chef-deploy

Monday, 20 July 2009

slide-75
SLIDE 75

deploy "/data/#{app}" do repo "git://server/path/app.git" branch "HEAD" user "myuser" enable_submodules true migrate true migration_command "rake db:migrate" environment "production" shallow_clone true revision '5DE77F8ADC' restart_command “...” role “myrole” action :deploy end

Monday, 20 July 2009

slide-76
SLIDE 76

Callbacks

  • Ruby scripts in your app’s deploy/
  • before_migrate, before_symlink,

before_restart, after_restart

  • Rails environment and ‘role’ passed as

arguments to callback

  • Could control this via

role node[:myapp][:role]

Monday, 20 July 2009

slide-77
SLIDE 77

Single source for gem dependencies

  • Specify gems in gems.yml in your app’s root
  • :name: foo

:version: "1.3"

  • :name: bar

:version: "2.0.1"

Monday, 20 July 2009

slide-78
SLIDE 78

Deployment strategy

  • Unlikely you want deploy to be attempted

with the default chef-client behavior

  • chef-deploy developed against a Chef Solo

world view: explicit execution

  • Use attribute to control deployment
  • Work in progress

Monday, 20 July 2009

slide-79
SLIDE 79

Gotchas

  • Chef-deploy assumes shared config/

database.yml

  • Usual package/gem conflicts
  • Don’t install rake from packages! (but

cookbooks are getting better at protecting you from this)

Monday, 20 July 2009

slide-80
SLIDE 80

Chef Solo

Monday, 20 July 2009

slide-81
SLIDE 81

Server-less operation

  • Bundle up the cookbooks in a tarball
  • Set attributes in a JSON file
  • Good to go!

Monday, 20 July 2009

slide-82
SLIDE 82

Deploying with solo

  • Tar up your cookbooks
  • Create a solo.rb

file_cache_path “/tmp/chef-solo” cookbook_path “/tmp/chef-solo/ cookbooks”

Monday, 20 July 2009

slide-83
SLIDE 83

Deploying with solo (2)

  • Create your JSON, e.g.

{ “recipes”: “chef-server”, “myvar”: “foo” }

  • Execute

chef-solo -c solo.rb -j chef.json

  • r http://path/to/tarball.tgz
  • JSON path can be URL too

Monday, 20 July 2009

slide-84
SLIDE 84

Why Chef Solo?

  • When you don’t or can’t control access to

the server

  • When clients aren’t in the same security

zone

  • When you care about installation rather

than long-term maintenance

Monday, 20 July 2009

slide-85
SLIDE 85

REST API

Monday, 20 July 2009

slide-86
SLIDE 86

Chef’s REST API

  • Chef’s REST API is pretty mature
  • Reused a lot internally
  • Best way to programmatically integrate
  • Chef wiki carries API examples

Monday, 20 July 2009

slide-87
SLIDE 87

What can you do with the API?

  • Programmatic access to the server
  • Add remove/recipes from nodes
  • Interrogate and set attributes
  • Perform searches

Monday, 20 July 2009

slide-88
SLIDE 88

API authentication

  • Register in the same way a node does

Chef::Config.from_file( “/etc/chef/server.rb”) @rest = Chef::REST.new( Chef::Config[:registration_url]) @rest.register(user, password)

  • Thereafter, authenticate

@rest.authenticate(user, password)

Monday, 20 July 2009

slide-89
SLIDE 89

Manipulating nodes

node = @rest.get_rest(“nodes/ foo_example_com”) puts node.recipes.inspect node.recipes << “apache2” puts node[:myattr].inspect node[:myattr] = { :foo => “bar” } @rest.put_rest(“nodes/foo_example_com”, node)

Monday, 20 July 2009

slide-90
SLIDE 90

Knife

  • Basic command line interface to the server
  • For now, get from http://gist.github.com/

104080

Monday, 20 July 2009

slide-91
SLIDE 91

Searching

Monday, 20 July 2009

slide-92
SLIDE 92

Searching the server

  • Powerful feature
  • Not that mature yet
  • Ferret indexes the Chef Server database
  • Queries expressed in FQL

Monday, 20 July 2009

slide-93
SLIDE 93

Access from recipes

  • search(INDEX, QUERY)
  • search(:node, “*”) reports every node in

the DB

  • Find the IP of every node running Apache

search(:node, “recipe:apache2”).collect {|n| n[‘ipaddress’]}

Monday, 20 July 2009

slide-94
SLIDE 94

Access from REST API

  • As implemented in the Web UI

@rest.get_rest( "search/node?q=recipe:apache2")

Monday, 20 July 2009

slide-95
SLIDE 95

Development patterns

Monday, 20 July 2009

slide-96
SLIDE 96

Git strategy

  • Use submodules to bring in 3rd party

cookbooks

  • Develop against testbed, push to shared

repository

  • Server install rule does a git pull

Monday, 20 July 2009

slide-97
SLIDE 97

VM testbed

  • Use a VM tool that supports snapshotting
  • VirtualBox is free
  • VMware good, supported by Poolparty
  • Use Avahi/Bonjour for convenience

Monday, 20 July 2009

slide-98
SLIDE 98

Use roles

  • Allow site-wide customization
  • Bundling your configuration with choice of

cookbooks

  • Recipes can then implement control

inflexion points using attributes

Monday, 20 July 2009

slide-99
SLIDE 99

Refactor into definitions & attributes

  • For maintainability, consider refactoring
  • bvious components into definitions
  • e.g. the directory creation stage of a Rails

app (what cap deploy:setup does)

Monday, 20 July 2009

slide-100
SLIDE 100

Chef & EC2

Monday, 20 July 2009

slide-101
SLIDE 101

In OpsCode cookbooks

  • ec2 cookbook
  • EC2 awareness in, e.g. mysql recipes
  • Bunch of handy EC2 attributes exposed

Monday, 20 July 2009

slide-102
SLIDE 102

Chef AMIs

  • Work in progress
  • Preconfigured Ubuntu with chef-client and/
  • r server
  • Chef attributes sent as instance data
  • Chef wiki has worked EC2 + Rails

architecture

Monday, 20 July 2009

slide-103
SLIDE 103

Poolparty

  • Configure and deploy to the cloud
  • Uses Chef
  • http://poolpartyrb.com/

Monday, 20 July 2009

slide-104
SLIDE 104

What Poolparty does

  • Launches VM (EC2 or VMware), waits for IP

and ssh

  • Bootstrap: rsyncs dependencies and installs
  • Configure: compile cookbooks, rsyncs,

executes Chef Solo

  • Verifies installation

Monday, 20 July 2009

slide-105
SLIDE 105

Community resources

  • Wiki is a great and ever-improving

reference http://wiki.opscode.com/display/chef/Home

  • IRC

irc://irc.freenode.net/chef

  • Mailing list

Monday, 20 July 2009

slide-106
SLIDE 106

The future

  • Chef is evolving rapidly
  • Platform support improving through

contributions

  • Opscode-agent
  • nanite
  • selective resource execution

Monday, 20 July 2009

slide-107
SLIDE 107

In conclusion

  • Please rate this tutorial and leave

comments http://bit.ly/chef-oscon

  • Q&A
  • Thank you!

Monday, 20 July 2009