StarCluster - NumPy/SciPy Computing on Amazons Elastic Compute Cloud - - PowerPoint PPT Presentation

▶

Aug 29, 2022 186 likes •944 views

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions StarCluster - NumPy/SciPy Computing on Amazons Elastic Compute Cloud (EC2) Justin Riley Software Tools for Academics and Researchers Office of Educational Innovation

SLIDE 1

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

Justin Riley

Software Tools for Academics and Researchers Office of Educational Innovation and Technology Massachusetts Institute of Technology

SciPy 2010

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 2

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Outline

1 Introduction

STAR Group Motivation behind StarCluster

2 Amazon EC2 Basics 3 StarCluster Overview

Features Configuration Quick Demo Customizing StarCluster Using Elastic Block Storage Creating Plugins

4 Conclusions

Future Work Where can I learn more?

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 3

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Outline

1 Introduction

STAR Group Motivation behind StarCluster

2 Amazon EC2 Basics 3 StarCluster Overview

Features Configuration Quick Demo Customizing StarCluster Using Elastic Block Storage Creating Plugins

4 Conclusions

Future Work Where can I learn more?

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 4

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group

Software Tools for Academics and Researchers

1 Work with faculty at MIT to develop software for

classroom/research

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 5

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group

Software Tools for Academics and Researchers

1 Work with faculty at MIT to develop software for

classroom/research

2 StarBiochem - Protein Visualization Tool

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 6

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group

Software Tools for Academics and Researchers

1 Work with faculty at MIT to develop software for

classroom/research

2 StarBiochem - Protein Visualization Tool 3 StarGenetics - Genetic cross-simulator

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 7

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group

Software Tools for Academics and Researchers

1 Work with faculty at MIT to develop software for

classroom/research

2 StarBiochem - Protein Visualization Tool 3 StarGenetics - Genetic cross-simulator 4 StarMolsim - Web-based MD/Quantum simulations

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 8

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group

Software Tools for Academics and Researchers

1 Work with faculty at MIT to develop software for

classroom/research

2 StarBiochem - Protein Visualization Tool 3 StarGenetics - Genetic cross-simulator 4 StarMolsim - Web-based MD/Quantum simulations 5 ... and more (http://web.mit.edu/star)

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 9

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster

Motivations for StarCluster. . .

Cluster Configuration is Hard Cluster Configuration

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 10

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster

Motivations for StarCluster. . .

Cluster Configuration is Hard Cluster Configuration Obtaining access to hardware can be a challenge

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 11

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster

Motivations for StarCluster. . .

Cluster Configuration is Hard Cluster Configuration Obtaining access to hardware can be a challenge Configuring and maintaing cluster configurations is hard

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 12

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster

Motivations for StarCluster. . .

Cluster Configuration is Hard Cluster Configuration Obtaining access to hardware can be a challenge Configuring and maintaing cluster configurations is hard Traditional resources = administrative overhead

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 13

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster

StarHPC. . .

User EC2 Virtual Desktop SSH VNC

Figure: StarHPC Overview

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 14

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster

StarMolSim. . .

User MolSim using GenePattern EC2 WEB SSH

Figure: StarMolSim Overview

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 15

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Outline

1 Introduction

STAR Group Motivation behind StarCluster

2 Amazon EC2 Basics 3 StarCluster Overview

Features Configuration Quick Demo Customizing StarCluster Using Elastic Block Storage Creating Plugins

4 Conclusions

Future Work Where can I learn more?

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 16

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Elastic Compute Cloud Overview

Infrastructure as a Service (IaaS) Cloud Computing Model

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 17

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Elastic Compute Cloud Overview

Infrastructure as a Service (IaaS) Cloud Computing Model Request up to 20 virtual machines by default

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 18

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Elastic Compute Cloud Overview

Infrastructure as a Service (IaaS) Cloud Computing Model Request up to 20 virtual machines by default Full root access via SSH

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 19

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Elastic Compute Cloud Overview

Infrastructure as a Service (IaaS) Cloud Computing Model Request up to 20 virtual machines by default Full root access via SSH Only pay for what you use

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 20

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Elastic Block Storage

Analagous to a ”Virtual USB pendrive”

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 21

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Elastic Block Storage

Analagous to a ”Virtual USB pendrive” Size can be anywhere from 1GB-1TB per volume

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 22

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Elastic Block Storage

Analagous to a ”Virtual USB pendrive” Size can be anywhere from 1GB-1TB per volume Supports snapshotting volumes to create backups

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 23

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Elastic Block Storage

Analagous to a ”Virtual USB pendrive” Size can be anywhere from 1GB-1TB per volume Supports snapshotting volumes to create backups Ability to create new volumes based on snapshots

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 24

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Standard Instances

Definition 1 Compute Unit (CU) = 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Instance Arch CPU (CU) RAM Storage I/O Cost/hr Small 32bit 1 (x1) 1.7GB 160GB Moderate $0.085 Large 64bit 2 (x2) 7.5GB 860GB High $0.34 X-Large 64bit 2 (x4) 15GB 1.69TB High $0.68

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 25

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

High-Memory Instances

Definition 1 Compute Unit (CU) = 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Instance Arch CPU (CU) RAM Storage I/O Cost/hr X-Large 64bit 3.25 (x2) 17.1GB 420GB Moderate $0.50 2X-Large 64bit 3.25 (x4) 34.2GB 850GB High $1.20 4X-Large 64bit 3.25 (x8) 68.4GB 1.69TB High $2.40

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 26

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

High-CPU Instances

Definition 1 Compute Unit (CU) = 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Instance Arch CPU (CU) RAM Storage I/O Cost/hr Medium 32bit 2.5 (x2) 1.7GB 160GB Moderate $0.17 X-Large 64bit 2.5 (x8) 15GB 1.69TB High $0.68

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 27

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

AWS Funding Opportunities

AWS In Education http://aws.amazon.com/education/

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 28

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

AWS Funding Opportunities

AWS In Education http://aws.amazon.com/education/ Teaching Grants for educators using AWS in courses

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 29

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

AWS Funding Opportunities

AWS In Education http://aws.amazon.com/education/ Teaching Grants for educators using AWS in courses Research Grants for academic researchers using AWS in their work

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 30

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

AWS Funding Opportunities

AWS In Education http://aws.amazon.com/education/ Teaching Grants for educators using AWS in courses Research Grants for academic researchers using AWS in their work Project Grants for student organizations pursuing entrepreneurial endeavors

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 31

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Questions?

?

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 32

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Outline

1 Introduction

STAR Group Motivation behind StarCluster

2 Amazon EC2 Basics 3 StarCluster Overview

Features Configuration Quick Demo Customizing StarCluster Using Elastic Block Storage Creating Plugins

4 Conclusions

Future Work Where can I learn more?

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 33

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

About

StarCluster allows anyone to create their own scientific computing cluster on Amazon’s Elastic Compute Cloud (EC2) Dependencies: Registered and fully configured EC2 account

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 34

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

About

StarCluster allows anyone to create their own scientific computing cluster on Amazon’s Elastic Compute Cloud (EC2) Dependencies: Registered and fully configured EC2 account Python 2.4+

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 35

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

About

StarCluster allows anyone to create their own scientific computing cluster on Amazon’s Elastic Compute Cloud (EC2) Dependencies: Registered and fully configured EC2 account Python 2.4+ Boto (AWS library for Python)

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 36

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

About

StarCluster allows anyone to create their own scientific computing cluster on Amazon’s Elastic Compute Cloud (EC2) Dependencies: Registered and fully configured EC2 account Python 2.4+ Boto (AWS library for Python) Paramiko (SSH library for Python)

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 37

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

StarCluster Features

Simple configuration file for defining cluster settings

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 38

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

StarCluster Features

Simple configuration file for defining cluster settings Single ”start” command to create a cluster

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 39

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

StarCluster Features

Simple configuration file for defining cluster settings Single ”start” command to create a cluster ”stop” command to terminate a cluster and stop paying for it

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 40

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

StarCluster Features

Simple configuration file for defining cluster settings Single ”start” command to create a cluster ”stop” command to terminate a cluster and stop paying for it Automatic configuration of:

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 41

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

StarCluster Features

Simple configuration file for defining cluster settings Single ”start” command to create a cluster ”stop” command to terminate a cluster and stop paying for it Automatic configuration of: Network File System (/home and all EBS volumes)

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 42

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

StarCluster Features

Simple configuration file for defining cluster settings Single ”start” command to create a cluster ”stop” command to terminate a cluster and stop paying for it Automatic configuration of: Network File System (/home and all EBS volumes) Sun Grid Engine

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 43

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

StarCluster Features

Simple configuration file for defining cluster settings Single ”start” command to create a cluster ”stop” command to terminate a cluster and stop paying for it Automatic configuration of: Network File System (/home and all EBS volumes) Sun Grid Engine Passwordless-ssh

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 44

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

StarCluster Features

Simple configuration file for defining cluster settings Single ”start” command to create a cluster ”stop” command to terminate a cluster and stop paying for it Automatic configuration of: Network File System (/home and all EBS volumes) Sun Grid Engine Passwordless-ssh OpenMPI, etc

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 45

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

NumPy/SciPy on StarCluster

Custom compiled Atlas/NumPy/SciPy for 8-core instance types

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 46

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Features

NumPy/SciPy on StarCluster

Custom compiled Atlas/NumPy/SciPy for 8-core instance types Custom NumPy/SciPy Cookbook: http://starcluster.scripts.mit.edu/ starcluster/wiki/index.php

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 47

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Configuration

Configuration

INI-based configuration file

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 48

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Configuration

Configuration

INI-based configuration file ”cluster templates” define cluster configuration

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 49

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Configuration

Example Config

1 [aws info ] 2 aws access key id = #your aws access key id 3 aws secret access key = #your secret access key 4 aws user id= #your userid 5 6 [key mykeypair] 7 key location =/home/myuser/.ssh/mykeypair.rsa 8 9 [ cluster smallcluster ] 10 cluster size = 2 11 keyname = gsg−keypair 12 cluster user = sgeadmin 13 cluster shell = bash 14 node image id = ami−d1c42db8 15 node instance type = m1.small

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 50

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Configuration

Extending Cluster Templates

Re-using cluster template settings:

1 .... 2 3 [ cluster largecluster ] 4 extends=smallcluster 5 cluster size =16 6 node image id = ami−a5c42dcc 7 node instance type = c1.xlarge

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 51

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Quick Demo

Brief Demo

1 $ starcluster start physicscluster 2 >>> Starting cluster ... 3 >>> Launching a 2−node cluster... 4 >>> Launching master node... 5 >>> Launching worker nodes... 6 >>> Waiting for cluster to start ... 7 >>> The master node is ec2−123−12−12−123.compute−1.amazonaws.com 8 >>> Attaching volume vol−99999999 to master node on /dev/sdz ... 9 >>> Setting up the cluster ... 10 >>> Mounting EBS volume vol−99999999 on /home... 11 >>> Creating cluster user: myuser 12 >>> Configuring scratch space for user: myuser 13 >>> Configuring /etc/hosts on each node 14 >>> Configuring NFS... 15 >>> Configuring passwordless ssh for user: myuser 16 >>> Installing Sun Grid Engine ...

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 52

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Customizing StarCluster

Customizing StarCluster

How do I install my own software? Launch a single instance using either 32/64bit StarCluster AMI

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 53

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Customizing StarCluster

Customizing StarCluster

How do I install my own software? Launch a single instance using either 32/64bit StarCluster AMI Login via ssh and install software

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 54

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Customizing StarCluster

Customizing StarCluster

How do I install my own software? Launch a single instance using either 32/64bit StarCluster AMI Login via ssh and install software Use starcluster’s ”createimage” command to create a new custom AMI

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 55

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Customizing StarCluster

Customizing StarCluster

How do I install my own software? Launch a single instance using either 32/64bit StarCluster AMI Login via ssh and install software Use starcluster’s ”createimage” command to create a new custom AMI Specify your new AMI id in the StarCluster configuration file

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 56

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Elastic Block Storage

Attached (mounted) to the master node

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 57

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Elastic Block Storage

Attached (mounted) to the master node NFS-shared to all nodes

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 58

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Elastic Block Storage

Attached (mounted) to the master node NFS-shared to all nodes All data written to EBS is persisted automatically

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 59

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Creating New EBS Volume with StarCluster

How do we create new EBS volumes?

1 $ starcluster createvolume 20 us−east−1d

This command automatically handles:

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 60

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Creating New EBS Volume with StarCluster

How do we create new EBS volumes?

1 $ starcluster createvolume 20 us−east−1d

This command automatically handles: Launching a ”host” instance

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 61

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Creating New EBS Volume with StarCluster

How do we create new EBS volumes?

1 $ starcluster createvolume 20 us−east−1d

This command automatically handles: Launching a ”host” instance Attaching the volume to the instance

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 62

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Creating New EBS Volume with StarCluster

How do we create new EBS volumes?

1 $ starcluster createvolume 20 us−east−1d

This command automatically handles: Launching a ”host” instance Attaching the volume to the instance Partitioning the entire volume into a single partition

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 63

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Creating New EBS Volume with StarCluster

How do we create new EBS volumes?

1 $ starcluster createvolume 20 us−east−1d

This command automatically handles: Launching a ”host” instance Attaching the volume to the instance Partitioning the entire volume into a single partition Formatting the volume with ext3 filesystem

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 64

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Using Elastic Block Storage

Creating New EBS Volume with StarCluster

How do we create new EBS volumes?

1 $ starcluster createvolume 20 us−east−1d

This command automatically handles: Launching a ”host” instance Attaching the volume to the instance Partitioning the entire volume into a single partition Formatting the volume with ext3 filesystem Terminating the host instance

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 65

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Creating Plugins

Plugin System

ubuntu.py

1 from starcluster . logger import log 2 from starcluster . clustersetup import ClusterSetup 3 4 class PackageInstaller (ClusterSetup ): 5 6 def init ( self , pkg to install ): 7 self . pkg to install = pkg to install 8 9 def run( self , nodes, master, user, user shell , volumes): 10 for node in nodes: 11

log. info (” Installing

%s on node: %s” % \ 12 ( self . pkg to install , node. alias )) 13 node.ssh.execute(’apt−get −y install %s’ % \ 14 self . pkg to install )

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 66

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Creating Plugins

Plugin Config

Enabling the ubuntu plugin in the config

1 [ plugin pkginstaller ] 2 setup class = ubuntu.PackageInstaller 3 pkg to install = htop

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 67

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Creating Plugins

Questions?

?

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 68

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions

Outline

1 Introduction

STAR Group Motivation behind StarCluster

2 Amazon EC2 Basics 3 StarCluster Overview

Features Configuration Quick Demo Customizing StarCluster Using Elastic Block Storage Creating Plugins

4 Conclusions

Future Work Where can I learn more?

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 69

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Future Work

Future Work

Dynamic Load Balancing via Sun Grid Engine

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 70

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Future Work

Future Work

Dynamic Load Balancing via Sun Grid Engine Supported plugins (ipcluster, mpi implementations, etc)

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 71

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Where can I learn more?

Questions and Answers

Want to know more? Homepage: http://web.mit.edu/starcluster

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 72

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Where can I learn more?

Questions and Answers

Want to know more? Homepage: http://web.mit.edu/starcluster Code: http://github.com/jtriley/StarCluster

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 73

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Where can I learn more?

Questions and Answers

Want to know more? Homepage: http://web.mit.edu/starcluster Code: http://github.com/jtriley/StarCluster Mailing list: http://web.mit.edu/stardev/cluster/mailinglist.html

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

SLIDE 74

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Where can I learn more?

Questions and Answers

Want to know more? Homepage: http://web.mit.edu/starcluster Code: http://github.com/jtriley/StarCluster Mailing list: http://web.mit.edu/stardev/cluster/mailinglist.html Software Tools for Academics and Researchers: http://web.mit.edu/star

Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)