Building Custom Linux Images Building Custom Linux Images for Amazon EC2 for Amazon EC2 Eric Hammond Eric Hammond VP Technology VP Technology CampusExplorer.com CampusExplorer.com O'Reilly OSCON Open Source Convention 2009 O'Reilly OSCON Open Source Convention 2009
Intro to Amazon EC2 Intro to Amazon EC2 Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 2
What is Amazon EC2? On Demand, Self Service, Pay As You Go, Computing Infrastructure Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 3
EC2 - Computing Infrastructure Servers (CPU, RAM, Disk) Static IP addresses Firewalls SAN – Storage Area Network Monitoring Load Balancing Auto Scaling Network bandwidth Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 4
EC2 Terminology Server “Instance” Static IP “Elastic IP Address” Firewall “Security Group” SAN “Elastic Block Store” Monitoring “CloudWatch” Load Balancing “Elastic Load Balancing” Auto Scaling “Auto Scaling” Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 5
EC2 – On Demand Scale up ...to meet demand Scale down ...to save money Scaling time measured in minutes Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 6
EC2 – Self Service AWS = Amazon “Web Services” No Amazon employees needed Complete root access to servers Write software to control EC2 Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 7
EC2 – Pay As You Go Pay for what you use Stop paying when you stop using it Server – pay by hour (wall clock) Network – pay by GB Storage – pay by GB/month Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 8
Amazon Machine Images (AMI) Amazon Machine Images (AMI) Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 9
Amazon Machine Images “image” = Copy of a boot disk Determines: Operating system (Linux, Windows) Distribution (Ubuntu, Debian, Fedora, ...) Release (Ubuntu 9.04 Intrepid) Software installed (LAMP, Rails, ...) Software configuration Should not include “data” Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 10
Image = Boot Disk Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 11
New EC2 Servers New EC2 server starts with a copy of the boot disk image Changes on the server do not affect the contents of the master image Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 12
New Server = New Copy of Disk Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 13
Where Do You Get an EC2 Image? Public EC2 images Building EC2 images from running systems Building EC2 images from scratch Building EC2 images using 3 rd party services Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 14
Public Linux Images Public Linux Images Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 15
Public EC2 Images Fedora Red Hat CentOS Ubuntu Debian OpenSuse Gentoo (OpenSolaris) (Windows 2003) Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 16
Needle In a Haystack Over 3,300 public EC2 images as of Jul 2009 Around 500 of those are mine About 60 of those are the latest release of their series Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 17
EC2 Images Available on Alestic.com Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 18
Advantages of Public Images Convenience Maintenance Support Community Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 19
Drawbacks of Public Images Trust, security, longevity Software setup Documentation Startup/setup time Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 20
Using Public EC2 Images Research images Research image publishers Stay current with latest images Install & configure software at bootup Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 21
user-data Startup Scripts Server startup scripts with user-data ec2userdata.notlong.com If user-data starts with #! then the server runs it as root on the first boot Supported by Ubuntu and Debian images Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 22
user-data startup script #!/bin/bash -x export DEBIAN_FRONTEND=noninteractive apt-get update && apt-get upgrade -y tasksel install lamp-server apt-get install -y wordpress php5-gd</dev/null META=http://169.254.169.254/2007-12-15/meta-data host=$(wget -qO- $META/public-hostname/) WP=/usr/share/doc/wordpress/examples perl -pi -e 's/^ping/true/' $WP/setup-mysql bash $WP/setup-mysql -n wordpress $host mv /var/www /var/www.orig ln -s /usr/share/wordpress /var/www /etc/init.d/apache2 restart Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 23
Building EC2 Images: Building EC2 Images: Concepts Concepts Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 24
Concepts Bucket – possibly public, globally unique name; contains multiple images Prefix – possibly public name, unique within bucket; identifies individual image Image ID (AMI ID) – Unique identifier for an individual image; assigned by EC2 Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 25
Access Identifiers 9+ access identifiers, some available on ec2access.notlong.com AWS user id (account id without dashes) Amazon.com account email, password AWS access key id, secret access key X.509 certificate, private key ssh public key, private key (“keypair”) Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 26
Building Images from Building Images from Running Systems Running Systems Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 27
Rebundle a Running EC2 Server Run an existing EC2 image Install and customize software “Rebundle” and upload Register as a new image Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 28
Rebundle a Running EC2 Server # Upload certificate+private key scp -i <KEYPAIR>.pem \ {cert,pk}-*.pem \ root@<HOST>:/tmp/ # Connect to the instance ssh -i <KEYPAIR>.pem root@<HOST> # Install and customize software ... Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 29
Rebundle a Running EC2 Server # Bundle the current file system sudo ec2-bundle-vol \ --prefix <PREFIX> \ --user <AWS_USER_ID> \ --arch i386 \ --destination /mnt \ --cert /tmp/cert-*.pem \ --privatekey /tmp/pk-*.pem \ --size 10240 \ --exclude /mnt,/tmp,/root/.ssh Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 30
Rebundle a Running EC2 Server # Upload the bundle to S3 ec2-upload-bundle \ --bucket <BUCKET> \ -m /mnt/<PREFIX>.manifest.xml \ --access-key <ACCESS_KEY_ID> \ --secret-key <SECRET_KEY> Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 31
Rebundle a Running EC2 Server # Register the image ec2-register \ <BUCKET>/<PREFIX>.manifest.xml Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 32
Free Tip Don't terminate the server until you have tested the new EC2 image Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 33
Advantages of Rebundling Seems easy Manual software configuration No special software/skills required Lots of tutorials available Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 34
Drawbacks of Rebundling Reproducibility Debug cycle Code branching Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 35
Building Images from Scratch Building Images from Scratch Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 36
Building From Scratch Create fresh copy of a boot file system ...in a subdirectory of the current file system Bundle & upload as the new image Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 37
Building From Scratch Build base OS (e.g., debootstrap) Install software packages ● Don't launch daemons on install Install kernel modules ● Special XFS kernel module for old kernel Install EC2 AMI command line tools /proc and /dev/pts file systems Eric Hammond Building Custom Linux Images for Amazon EC2 Alestic.com 38
Recommend
More recommend
