Grid on Demand Willem Toorop and Alain van Hoof { wtoorop,ahoof } - - PowerPoint PPT Presentation

Welcome

Grid on Demand

Willem Toorop and Alain van Hoof

{wtoorop,ahoof}@os3.nl

June 30, 2010

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 1 / 39

Introduction Research Question

Research Question

Can Grid computing be offered as a Cloud service and what type of applications benefit from this?

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 2 / 39

Introduction The Grid and The Cloud

The Grid and The Cloud: Sharing Resources

Sharing Resources to minimizing idle resources Resource Capacity Goods Processor Flops Compute Hard-disk memory Bytes Storage Network Bytes/sec Communication Bandwidth Grid Cloud (IaaS) Distribution By Collaboration By Cloud Provider Resource Type Heterogeneous Homogeneous Abstraction Middleware (open standards) Virtualization Sharing Guarantee Security infrastructure Financial infra

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 3 / 39

Introduction The Grid and The Cloud

The Grid and The Cloud: Dividing the Resources

Cloud Grid Grid Grid User User User VO User User VO User User

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 4 / 39

A Grid in the Cloud Urgent Computing

A Grid in The Cloud: Urgent Computing

GRID

• n

Demand

Using the elasticity and scalability of the Cloud and providing a Grid interface, current e-Science (computationally intensive science) applications can use a Grid on Demand when there is a high demand for resources for a short period of time. Example UrbanFlood (http://urbanflood.eu) wants to calculate flooding predictions when actual flooding happens, Grid on Demand can provide this.

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 5 / 39

A Grid in the Cloud Grid on Demand

Grid on Demand

Compute Nodes Head Node Job Submission Client

OS: Ubuntu 10.04 Grid: Globus 4.2.1 PBS: Torque 2.6.8

Grid Interface: Globus Toolkit 4.2 (open standard, widely used) Queue Manager (Distribution of jobs to nodes): Torque 2.6.2 Amazon EC2 Cloud Provider (de-facto standard with open API)

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 6 / 39

Implementation What makes the Grid, the Grid

What makes the Grid, the Grid

Typical Grid usage example spanning multiple organizations

Grid User Scheduler Storage Compute Compute Organization A Organization B Organization D Organization C store data use my data to compute X use 1st 1

2 of data

to compute 1

2X

use 2nd 1

2 of data

to compute 1

2X

access data a c c e s s d a t a

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 7 / 39

Implementation What makes the Grid, the Grid

Proxy certificates for delegation

The Grid is a collaboration between multiple organizations. Grid users need to be authorized to use remote resources Those resources should be authorized to perform actions

• n behalf of the Grid user

Proxy certificates enable delegation of authority

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 8 / 39

Implementation What makes the Grid, the Grid

Grid user create proxy Scheduler create cert. request Compute c

• m

p u t e X sign this Okay compute X

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation What makes the Grid, the Grid

Grid user create proxy Scheduler create cert. request Compute c

• m

p u t e X sign this Okay compute X

Virtual Organizations are based on delegation by proxy certificates.

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation What makes the Grid, the Grid

Grid user create proxy Scheduler create cert. request Compute c

• m

p u t e X sign this Okay compute X

Proxy certificates allow for dynamic resource allocation, But the hosts still need a CA signed certificate. This is a problem when hostname’s are assigned dynamically!

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation What makes the Grid, the Grid

Grid user create proxy Scheduler create cert. request Compute c

• m

p u t e X sign this Okay compute X

Proxy certificates allow for dynamic resource allocation, But the hosts still need a CA signed certificate. This is a problem when hostname’s are assigned dynamically! Fixed CA in image?

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation What makes the Grid, the Grid

Grid user create proxy Scheduler create cert. request Compute c

• m

p u t e X sign this Okay compute X

Proxy certificates allow for dynamic resource allocation, But the hosts still need a CA signed certificate. This is a problem when hostname’s are assigned dynamically! Fixed CA in image? Dynamic CA on every Launch?

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation Amazon Elastic Block Store

Easy Image management with Amazon Elastic Block Store

(A, S)

Pending

Running (IA, VS, H)

Stopping

Stopped (IA, VS)

Pending Shutting-down

Terminated IA (A′, S′) launch instance IA VS ← S stop start terminate t e r m i n a t e create AMI A′ S′ ← VS configure

Life cycle of Amazon Machine Images using Elastic Block Storage A = Amazon Machine Image, S = Snapshot, I = Instance, V = Volume, H = Hostname

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 10 / 39

Implementation EBS for configuration stages

EBS for configuration stages

(A, S) Running (IA, VS, H) Stopped (IA, VS) (A′, S′) Stopped (IA, VS) (A′′, S′′) Running (IA, VS, H) Stopped (IA, VS) (A′′′, S′′′) launch instance IA VS ← S shutdown for reuse start create AMI A′ S′ ← VS create AMI A′′ S′′ ← VS create AMI A′′′ S′′′ ← VS shutdown for sharing shutdown for sharing shutdown for reuse start provide Amazon Access Credentials

(configure cluster)

launch IA′ VS ← S′ launch IA′′′ VS ← S′′′ configure configure

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 11 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 12 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 13 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 14 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 15 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 16 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 17 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 18 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 19 / 39

Web Interface Configuring the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 20 / 39

Web Interface Configuring the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 21 / 39

Web Interface Configuring the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 22 / 39

Web Interface Configuring the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 23 / 39

Web Interface Real Grid certificate usage Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 24 / 39

Web Interface Real Grid certificate usage Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 25 / 39

Web Interface Real Grid certificate usage Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 26 / 39

Web Interface Elasticity of the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 27 / 39

Web Interface Elasticity of the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 28 / 39

Web Interface Elasticity of the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 29 / 39

Web Interface Saving your settings Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 30 / 39

Web Interface Saving your settings Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 31 / 39

Web Interface Saving your settings Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 32 / 39

“Real use” and Performance WS-VLAM and WAVE

“Real use” and Performance test

To test the possibilities and performance of Grid on Demand a Biomedical Application and a workflow manager are used

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39

“Real use” and Performance WS-VLAM and WAVE

“Real use” and Performance test

To test the possibilities and performance of Grid on Demand a Biomedical Application and a workflow manager are used Biomedical Application: WAVE Parallel (MonteCarlo simulation) application

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39

“Real use” and Performance WS-VLAM and WAVE

“Real use” and Performance test

To test the possibilities and performance of Grid on Demand a Biomedical Application and a workflow manager are used Biomedical Application: WAVE Parallel (MonteCarlo simulation) application Workflow Manager A (graphical) tool to assist complex e-Science application creation Creates a series of jobsubmissions WS-VLAM created by UvA SNE Group Connects to Globus Grid Interface

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39

“Real use” and Performance Connecting the Grid on Demand to a Workflow Manager

Connecting the Grid on Demand to a Workflow Manager

Compute Nodes Head Node

OS: Ubuntu 10.04 Grid: Globus 4.2.1 PBS: Torque 2.6.8

WAVE Biomedical Compute Application

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 34 / 39

“Real use” and Performance Preliminary performance results: total execution time

Preliminary performance results: total execution time

Grid on demand compared to a 32 node physical UvA cluster (DAS3) with Globus Grid interface Grid Jobs Total execution time DAS3 28 00:42:58 Grid on Demand 28 00:46:22 DAS3 98 02:46:36 Grid on Demand 98 00:52:26

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 35 / 39

“Real use” and Performance Preliminary performance results: 30 Jobs

Preliminary performance results of job pending and running time: 30 jobs

00:00:00 00:10:00 00:20:00 00:30:00 00:40:00 00:50:00 01:00:00 01:10:00 01:20:00 5 10 15 20 25 30 Time (HH:MM:SS) Job Number Performance on DAS3 - 30 Jobs Running time Pending time 00:00:00 00:10:00 00:20:00 00:30:00 00:40:00 00:50:00 01:00:00 01:10:00 01:20:00 5 10 15 20 25 30 Time (HH:MM:SS) Job Number Performance on Grid on Demand 30 Jobs Running time Pending time

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 36 / 39

“Real use” and Performance Preliminary performance results: 100 Jobs

Preliminary performance results of job pending and running time: 100 jobs

00:00:00 00:15:00 00:30:00 00:45:00 01:00:00 01:15:00 01:30:00 01:45:00 02:00:00 02:15:00 02:30:00 10 20 30 40 50 60 70 80 90 100 Time (HH:MM:SS) Job Number Performance on DAS3 - 100 Jobs Running time Pending time 00:00:00 00:15:00 00:30:00 00:45:00 01:00:00 01:15:00 01:30:00 01:45:00 02:00:00 02:15:00 02:30:00 20 40 60 80 100 Time (HH:MM:SS) Job Number Performance on Grid on Demand - 100 Jobs Running time Pending time

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 37 / 39

Conclusion

Conclusion

Grid on Demand can offer Grid as a Cloud service Parallel applications run/scale very well on Grid on Demand Grid on Demand can be a solution for Urgent Computing demands

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 38 / 39

The End

Questions

Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 39 / 39