Cloud... so much more than a tools fest Qcon London 2012 Patrick - - PowerPoint PPT Presentation

▶
cloud so much more than a tools fest
SMART_READER_LITE
LIVE PREVIEW

Cloud... so much more than a tools fest Qcon London 2012 Patrick - - PowerPoint PPT Presentation

Mar 06, 2024 138 likes •1.02k views

Cloud... so much more than a tools fest Qcon London 2012 Patrick Debois http://localizationlocalisation.files.wordpress.com/2010/08/rolling-out-old-tools.jpg Vagrant & http://jedi.be/blog Veewee @patrickdebois Freelancer Context: a

slide-1
SLIDE 1

Cloud... so much more than a tools fest

Patrick Debois

http://localizationlocalisation.files.wordpress.com/2010/08/rolling-out-old-tools.jpg

Qcon London 2012

slide-2
SLIDE 2

Vagrant & Veewee http://jedi.be/blog

slide-3
SLIDE 3

@patrickdebois

slide-4
SLIDE 4

Freelancer

slide-5
SLIDE 5

Context: a traditional enterprise

slide-6
SLIDE 6

http://1.bp.blogspot.com/-dJydLIEZCQA/TtO26OpjywI/AAAAAAAADBg/wo3Vpl-eesc/s800/Cat%2BArmy.jpg

With a developer group

  • n a mission

to innovate

slide-7
SLIDE 7

Realtime TV - Web interaction

slide-8
SLIDE 8

Second Screen Applications

slide-9
SLIDE 9

http://www.twylah.com/billycoover/topics/xamarin

slide-10
SLIDE 10

http://icanhascheezburger.files.wordpress.com/2011/09/2afc3f14-2ab1-49dc-819e-87490c58b16f.jpg

TV-show +/- 1 million concurrent viewers

slide-11
SLIDE 11

http://www.twylah.com/billycoover/topics/xamarin

slide-12
SLIDE 12
slide-13
SLIDE 13

Google App Engine

slide-14
SLIDE 14

Abstraction is AWESOME

slide-15
SLIDE 15

If it fails hard to debug/ understand

slide-16
SLIDE 16

Too generic

Esper Python Nodejs Redis Mongodb Hadoop Java Rails Mysql ETL tools

slide-17
SLIDE 17

Need more control

slide-18
SLIDE 18

IAAS

http://3.bp.blogspot.com/-TgmC6DkVNiU/TzCTP_BxWqI/AAAAAAAAAb8/797iiSLQ3FY/s1600/funny-pictures-cat-fixes-your-computer.jpg

slide-19
SLIDE 19

http://meowcheese.com/files/lolpics/2010/05/dog-not-cat.jpg

I never did any cloud before. Only enterprise stuff

slide-20
SLIDE 20

http://www.collthings.co.uk/2008/06/10-very-rare-clouds.html

What I learned from managing Production Servers

slide-21
SLIDE 21

WEB APP DB A Single Server

slide-22
SLIDE 22

WEB APP DB Starts Growing WEB APP WEB APP

slide-23
SLIDE 23

DB Generic Servers become Specific Servers WEB APP WEB APP WEB APP

slide-24
SLIDE 24

DB Physical becomes Virtual WEB APP WEB APP WEB APP xen/vsphere/kvm/...

slide-25
SLIDE 25

DB We learned cloning isn’t working WEBvI APP WEBv2 APP WEBv1 APP

slide-26
SLIDE 26

We introduce config management APP WEB DB = = = WEB JEOS Config APP JEOS Config DB JEOS Config

slide-27
SLIDE 27

Infrastructure as code WEB JEOS Config APP JEOS Config DB JEOS Config Config Management Code Repository Cfengine/Puppet/Chef

slide-28
SLIDE 28

http://www.collthings.co.uk/2008/06/10-very-rare-clouds.html

What I learned from managing Test Servers

slide-29
SLIDE 29

Infrastructure Code Repository DB WEB APP DB WEB APP TEST PROD Reuse Across Environments DEV

slide-30
SLIDE 30

TEST PROD Explosion of VM creation

DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Infrastructure Code Repository Application Code Repository

DB WEB APP DB WEB APP DB WEB APP DB WEB APP

DEV

slide-31
SLIDE 31

UI Interface to Automated Provisioning of VMS

INFRA

DB WEB APP

APP

DB WEB APP

VM VM VM Cobbler,Spacewalk, ...

slide-32
SLIDE 32

INFRA

DB WEB APP

APP

DB WEB APP

VM VM VM Config Management Puppet/Chef Server Metadata Registry for Systems

slide-33
SLIDE 33
slide-34
SLIDE 34
slide-35
SLIDE 35

I’VE NEVER INSTALLED A GUI FOR THAT.

slide-36
SLIDE 36

$ knife ec2 server create \

  • r 'role[webserver]' -I ami-7000f019 \
  • f m1.small \
  • A 'Your AWS Access Key ID' \
  • K 'Your AWS Secret Access Key'
slide-37
SLIDE 37

$ ¡puppet ¡node ¡create ¡-­‑-­‑image ¡ami-­‑XxXXxXXX ¡\

  • ­‑-­‑keypair ¡puppetlabs.admin ¡-­‑-­‑type ¡m1.small

aka “cloudpack”

slide-38
SLIDE 38

http://fog.io @geemus Servers Keys Elastic Load balancers Security groups Custom Scripts S3 Storage E-Mail service

slide-39
SLIDE 39

AWS Network Constraints Security Groups only on creation Only 1 network interface ELB not on non- standard ports

http://iruntheinternet.com/lulzdump/images/cat-in-jar-cok-stuck-1298144751i.jpg

slide-40
SLIDE 40

Dynamic Monitoring

nodes ¡= ¡search(:node, ¡"hostname:[* ¡ TO ¡*] ¡AND ¡ chef_environment:#{node.chef_enviro nment}")

slide-41
SLIDE 41

http://icanhascheezburger.files.wordpress.com/2009/09/funny-pictures-cat-is-squished.jpg

VM creation failure, network hickups, disk erratic behavior

slide-42
SLIDE 42

Embrace

slide-43
SLIDE 43

Re-architect

slide-44
SLIDE 44

“Quis custodiet ipsos Custodes”

who watches the watchers

slide-45
SLIDE 45

Noops

slide-46
SLIDE 46

What I learned working in the (Amazon)cloud

slide-47
SLIDE 47

AMI JEOS Config Mgmt Config Mgmt VM Xen Web,App,DB Web,App,DB UI Provision AWS Console Internal Cloud what’s the difference ? Metadata Registry Metadata Registry

slide-48
SLIDE 48

AWS Console Fog Jclouds Boto AWS API From console to API Web UI “Internals” API Abstracted API

slide-49
SLIDE 49

Beyond Servers Components Server(s) Loadbalancers DNS Service IP Address Email Service EBS Volume Firewall Keys +

slide-50
SLIDE 50

Cloudformation

From server to stack Json file specifying

  • rder of component creation

and dependencies

slide-51
SLIDE 51

http://www.collthings.co.uk/2008/06/10-very-rare-clouds.html

What I learned from managing development Servers

slide-52
SLIDE 52

DB WEB APP Development moves from host into virtual machines HOST DB WEB APP HOST VM

slide-53
SLIDE 53

Development starts using config mgt DB WEB APP HOST VM JEOS DB WEB APP HOST VM Config

slide-54
SLIDE 54

Infrastructure Code Repository DB WEB APP DB WEB APP DB WEB APP DEV TEST PROD Reuse “code” across Environments

slide-55
SLIDE 55

DB WEB APP HOST VM JEOS http://vagrantup.com/

Simple CLI

UP PROVISION HALT DESTROY

Vagrantfile Config Basebox Puppet/Chef VM Management Vagrant

slide-56
SLIDE 56

INFRA

DB WEB APP

DEV TEST PROD Integrate with Continuous Integration

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Infrastructure Code Repository Application Code Repository

slide-57
SLIDE 57

DB WEB APP VM JEOS Config Development/VM workflow UP PROVISION HALT DESTROY

slide-58
SLIDE 58

INFRA

DB WEB APP

TEST PROD Peak Capacity needed in Prod

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

CLOUD

Setup Outgrew Test Lab Setup Outgrew VM on Laptop DEV

slide-59
SLIDE 59

Mccloud (simple CLI)

AWS

Fog Library (API)

JEOS Config Mgmt Web,App,DB AWS JEOS Config Mgmt Web,App,DB AWS JEOS Config Mgmt Web,App,DB DEV TEST PROD

Reuse “workflow” across Environments “If it’s hard to it more often”

slide-60
SLIDE 60

+Reuse workflow across hypervisors

Virtualbox KVM AWS Fog Library

JEOS Config Mgmt Web,App,DB JEOS Config Mgmt Web,App,DB JEOS Config Mgmt Web,App,DB customers

slide-61
SLIDE 61

Cloud Libs (Jclouds/Fog/Boto) embracing old and personal

AWS Rackspace Eucalyptus Openstack

“old” “new”

Vsphere Virtualbox Kvm Libvirt VM Fusion

“personal” “hybrid”

slide-62
SLIDE 62

http://github.com/jedi4ever/veewee Abstracting OS Installation Kickstart Preseed Unattended.xml Debian Ubuntu Solaris Win Archlinux Suse Centos Redhat Virtualbox KVM Fusion Parallels

slide-63
SLIDE 63

Cloud Libs (Jclouds/Fog/Boto) beyond servers

DNS Keys IPs Storage Loadbalancer Securitygroups

slide-64
SLIDE 64

Workflow beyond servers

up provision halt destroy ip balance sorry VM Other

slide-65
SLIDE 65

Self Servicing

slide-66
SLIDE 66

INFRA

DB WEB APP

DEV TEST PROD Continuous Integration to Continuous Delivery

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Infrastructure Code Repository Application Code Repository

Faster/Delivery Faster/Feedback

slide-67
SLIDE 67

Confidence

slide-68
SLIDE 68

Faster/Delivery Faster/Feedback DEV OPS

slide-69
SLIDE 69

Infrastructure ~ Code

slide-70
SLIDE 70

Add Test Watch Test Fail Run tests Write Code Refactor TDD Cycle

slide-71
SLIDE 71

Testing ~ Monitoring

Cucumber-nagios

slide-72
SLIDE 72

Debugging ~ Metrics

slide-73
SLIDE 73

TEST PROD Metrics Reuse

DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Collectd,Ganglia, Graphite, Opentsdb Traditional Ops

slide-74
SLIDE 74

TEST PROD Extend “metrics” to Development

DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Collectd,Ganglia, Graphite, Opentsdb DEV

DB WEB APP DB WEB APP

slide-75
SLIDE 75

TEST PROD Extend “logs” to Development

DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Logstash, Graylog DEV

DB WEB APP DB WEB APP

slide-76
SLIDE 76

TEST PROD Selfservicing “metrics injection”

DB WEB APP DB WEB APP DB WEB APP DB WEB APP

StatsD DEV

DB WEB APP DB WEB APP

slide-77
SLIDE 77

TEST PROD Selfservicing “alerts”

DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Tatle DEV

DB WEB APP DB WEB APP

slide-78
SLIDE 78

TEST PROD Selfservicing “graphs”

DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Graphite DEV

DB WEB APP DB WEB APP

slide-79
SLIDE 79

Repeating Service Pattern Simple API/CLI , Self-Servicing

Heroku Blitz.io New Relic App Deployment Load Testing App Metrics Logging Loggly Pager Duty Alerting Monitoring Pingdom Continuous Integration Cloudbees DatadogHQ App Metrics

slide-80
SLIDE 80

+Reuse workflow across monitoring tools

Nagios Zenoss Sensu “my dream” Library Collectd Ganglia Graphite

slide-81
SLIDE 81

Workflow reuse

Monitoring UP

slide-82
SLIDE 82

Abstracting

It’s all events

Timestamp - Key - Value Logs Metrics Monitoring Meta Ops

slide-83
SLIDE 83

Social IT

radiate information to where it’s needed

Business ‘Pulse’

slide-84
SLIDE 84

http://www.collthings.co.uk/2008/06/10-very-rare-clouds.html

So maybe you don’t have unlimited resources

slide-85
SLIDE 85

But you can shape your internal IT as a cloud

slide-86
SLIDE 86

Questions?

slide-87
SLIDE 87

http://www.collthings.co.uk/2008/06/10-very-rare-clouds.html

Thank you!