Grid on Demand Willem Toorop and Alain van Hoof { wtoorop,ahoof } - PowerPoint PPT Presentation

Welcome Grid on Demand Willem Toorop and Alain van Hoof { wtoorop,ahoof } @os3.nl June 30, 2010 Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 1 / 39

Introduction Research Question Research Question Can Grid computing be offered as a Cloud service and what type of applications benefit from this? Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 2 / 39

Introduction The Grid and The Cloud The Grid and The Cloud: Sharing Resources Sharing Resources to minimizing idle resources Resource Capacity Goods Processor Flops Compute Hard-disk memory Bytes Storage Network Bytes/sec Communication Bandwidth Grid Cloud (IaaS) Distribution By Collaboration By Cloud Provider Resource Type Heterogeneous Homogeneous Abstraction Middleware (open standards) Virtualization Sharing Guarantee Security infrastructure Financial infra Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 3 / 39

Introduction The Grid and The Cloud The Grid and The Cloud: Dividing the Resources VO Grid User User Cloud Grid Grid User User User User User VO Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 4 / 39

A Grid in the Cloud Urgent Computing A Grid in The Cloud: Urgent Computing GRID on Demand Using the elasticity and scalability of the Cloud and providing a Grid interface, current e-Science (computationally intensive science) applications can use a Grid on Demand when there is a high demand for resources for a short period of time. Example UrbanFlood (http://urbanflood.eu) wants to calculate flooding predictions when actual flooding happens, Grid on Demand can provide this. Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 5 / 39

A Grid in the Cloud Grid on Demand Grid on Demand Job Submission Client OS: Ubuntu 10.04 Grid: Globus 4.2.1 PBS: Torque 2.6.8 Compute Nodes Head Node Grid Interface: Globus Toolkit 4.2 (open standard, widely used) Queue Manager (Distribution of jobs to nodes): Torque 2.6.2 Amazon EC2 Cloud Provider (de-facto standard with open API) Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 6 / 39

Implementation What makes the Grid, the Grid What makes the Grid, the Grid Typical Grid usage example spanning multiple organizations Organization B 2 of data Compute use 1 st 1 access to compute 1 2 X data Organization C Organization A use my data to compute X store data Grid Storage Scheduler User use 2 nd 1 a t a d s s to compute 1 2 of data e c c a 2 X Compute Organization D Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 7 / 39

Implementation What makes the Grid, the Grid Proxy certificates for delegation The Grid is a collaboration between multiple organizations. Grid users need to be authorized to use remote resources Those resources should be authorized to perform actions on behalf of the Grid user Proxy certificates enable delegation of authority Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 8 / 39

Implementation What makes the Grid, the Grid Grid user create proxy Scheduler c o m p u t e X create cert. request sign this Compute Okay compute X Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation What makes the Grid, the Grid Grid user create proxy Scheduler c o m p u t e X create cert. request sign this Compute Okay compute X Virtual Organizations are based on delegation by proxy certificates. Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation What makes the Grid, the Grid Grid user Proxy certificates allow for dynamic resource allocation, create proxy Scheduler But the hosts still need a CA signed certificate. c o m p u t e X This is a problem when hostname’s create cert. are assigned dynamically! request sign this Compute Okay compute X Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation What makes the Grid, the Grid Grid user Proxy certificates allow for dynamic resource allocation, create proxy Scheduler But the hosts still need a CA signed certificate. c o m p u t e X This is a problem when hostname’s create cert. are assigned dynamically! request sign this Compute Fixed CA in image? Okay compute X Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation What makes the Grid, the Grid Grid user Proxy certificates allow for dynamic resource allocation, create proxy Scheduler But the hosts still need a CA signed certificate. c o m p u t e X This is a problem when hostname’s create cert. are assigned dynamically! request sign this Compute Fixed CA in image? Okay compute X Dynamic CA on every Launch? Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

Implementation Amazon Elastic Block Store Easy Image management with Amazon Elastic Block Store configure launch instance I A Running ( A , S ) Pending ( I A , V S , H ) terminate V S ← S stop Shutting-down Stopping Pending Terminated start e t I A a n i m r e t create AMI A ′ Stopped ( A ′ , S ′ ) ( I A , V S ) S ′ ← V S Life cycle of Amazon Machine Images using Elastic Block Storage A = Amazon Machine Image, S = Snapshot, I = Instance, V = Volume, H = Hostname Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 10 / 39

Implementation EBS for configuration stages EBS for configuration stages configure configure provide Amazon Access Credentials launch instance I A Running Running (configure cluster) ( A , S ) ( I A , V S , H ) ( I A , V S , H ) V S ← S shutdown shutdown for sharing for sharing shutdown shutdown for reuse for reuse start start launch I A ′ launch I A ′′′ V S ← S ′ V S ← S ′′′ Stopped Stopped Stopped ( I A , V S ) ( I A , V S ) ( I A , V S ) create AMI A ′ create AMI A ′′ create AMI A ′′′ S ′ ← V S S ′′ ← V S S ′′′ ← V S ( A ′ , S ′ ) ( A ′′ , S ′′ ) ( A ′′′ , S ′′′ ) Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 11 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 12 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 13 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 14 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 15 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 16 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 17 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 18 / 39

Web Interface Launching Grid on Demand Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 19 / 39

Web Interface Configuring the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 20 / 39

Web Interface Configuring the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 21 / 39

Web Interface Configuring the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 22 / 39

Web Interface Configuring the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 23 / 39

Web Interface Real Grid certificate usage Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 24 / 39

Web Interface Real Grid certificate usage Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 25 / 39

Web Interface Real Grid certificate usage Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 26 / 39

Web Interface Elasticity of the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 27 / 39

Web Interface Elasticity of the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 28 / 39

Web Interface Elasticity of the cluster Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 29 / 39

Web Interface Saving your settings Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 30 / 39

Web Interface Saving your settings Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 31 / 39

Web Interface Saving your settings Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 32 / 39

“Real use” and Performance WS-VLAM and WAVE “Real use” and Performance test To test the possibilities and performance of Grid on Demand a Biomedical Application and a workflow manager are used Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39