A Grid Workflow Infrastructure GGF10 Dieter Cybok Berlin, Germany - - PowerPoint PPT Presentation

Dieter Cybok, msg systems

1

A

A Grid Workflow Infrastructure

Dieter Cybok Consultant msg systems GGF10 Berlin, Germany Tuesday, March 9th, 2004

Dieter Cybok, msg systems

2

A

Motivation

• Scientific applications often require the creation of complex

collaborative workflows

• Many e-Scientists lack the necessary low-level expertise to

utilize the current generation of Grid toolkits

• Documented workflow specification is beneficial for modelling

and managing scientific processes; processes can be easily reused, modified and shared

Making live of e-Scientists easier

Dieter Cybok, msg systems

3

A

Grid Workflow - Patterns

• Workflow Patterns

– Reusing results from workflow research – Defining relevant set of workflow patterns relevant to e-Science applications

B A B A B A A

sequence parallel split loop multi choice

Dieter Cybok, msg systems

4

A

Grid Workflow - Approaches

• Orchestration

– Describes business logic and execution order – Executable processes – One central workflow engine

activity 1 activity 2 activity 3 activity 4 activity 5 activity 6 flow

W S D L W S D L

Grid service Grid service Grid service Grid service

Dieter Cybok, msg systems

5

A

Grid Workflow - Approaches

• Choreography

– Sequence of messages that involve multiple services – Public message exchanges that occur between Grid services – Services involved describe the part they play in the interaction

Grid service Grid service Grid service Grid service

Dieter Cybok, msg systems

6

A

Grid Workflow - Requirements

• Web services workflow management requirements:

– Managing transactional integrity – Compensating transactions – Managing exceptions

• Additional grid workflow requirements:

– Dealing with large amounts of data – Life cycle management

Dieter Cybok, msg systems

7

A

Grid Workflow Infrastructure Grid Workflow Execution Language Workflow Engine GT3 Technology Grid Workflow Infrastructure

Dieter Cybok, msg systems

8

A

Grid Workflow Execution Language

• We have considered both, BPEL4WS and WSCI as the base for

GWEL

• Reasons for choosing orchestration/BPEL4WS:

– Definition of end-to-end processes – Existing Grid services can be used – Central workflow engine

• GWEL:

– XML based – Elements and concepts of BPEL4WS are reused

Dieter Cybok, msg systems

9

A

Grid Workflow Execution Language

List of Data Sources: Name, Handle List of Variables: Name, messageType

Fault Handling Model

List of Fault Handlers: Name, Variable Activity Model

Instance Lifecycle Model

List of Instance Creators: Name, Factory List of Events: Type, PortTypeName, Operation Activity Model Control Flow

Activity Model

List of Activities: Name, portType, operation, variable, data source, data sink Instance Creation Events

GWEL definition

Name, Target Namespace

Factory Model

List of Factories: Name, Handle

Data Model Variables

Event Handling

Dieter Cybok, msg systems

10

A

Workflow Engine

• Architecture is based on the

workflow reference model

• Prototypical implementation

with Java

service 1 service 2 service 3

workflow engine

data base client control and small amounts of data data base data base data base GWEL file

Dieter Cybok, msg systems

11

A

Underlying Technology

• OGSA

– Concept for a framework of services that support Grid functionalities

• OGSI

– Technical specification of the concepts described in OGSA – GWI operates on OGSI-based Grid services

• GT3

– Implementation of OGSI and OGSA – GWI operates on GT 3.0

Dieter Cybok, msg systems

12

A

Case Study

• Obtaining Bayesian Networks form data

– Using the unweighted L1 metric spanning tree algorithm – Computationally expensive

• Workflow:

1. L1 service reads in data 2. Unweighted L1 metric measure is computed 3. Intermediate results are stored 4. Sort service reads in intermediate results 5. Intermediate results are sorted 6. Sorting final results

L1 service sort service

workflow engine

data base client data base data base 1 2 3 4 5 6 GWEL file

Dieter Cybok, msg systems

13

A

Case Study – GWEL document

<workflow name="Simple_Workflow" …> <factoryLinks> … </factoryLinks> <dataLinks> … </dataLinks> <lifecycle> <createInstance instance_name="l1_instance"> <factoryLink name="L1"/> </createInstance> … <eventHandlers> <onNotification instance_name="l1_instance" portType="NotificationL1PortType"

• peration="computeL1">

<destroyInstance instance_name="l1_instance"> <factoryLink name="L1"/> </destroyInstance> </onNotification> … </eventHandlers> </lifecycle> <controlflow> <sequence name="sequence1"> <invoke instance_name="l1_instance" portType="l1_port"

• peration="computeL1"

dataInFrom="raw_data_input_db" dataOutTo= "l1_result_db"/> … </sequence> </controlflow> </workflow>

workflow.gwel

Dieter Cybok, msg systems

14

A

Case Study – Activity Diagram

Client Workflow Engine factory Workflow Engine instance L1 factory L1 instance Sort factory Sort instance submit GWEL document call workflow factory create workflow engine instance parse GWEL document return GSH to the client store L1 instance GSH store Sort instance GSH invoke L1 instance invoke Sort instance destroy L1 instance destroy Sort instance create L1 service instance return GSH to the workflow engine instance input data, compute it and store it send notification create Sort service instance return GSH to the workflow engine instance input data, compute it and store it send notification

client WFE fact. WFE instance L1 instance L1 factory Sort factory Sort instance

Dieter Cybok, msg systems

15

A

Conclusions

• e-Scientists need a generally applicable grid workflow

infrastructure

– Specifying processes graphically – Locating available services easily – Plugging together services automatically

• Our solution

– Prototypical infrastructure – Feasibility study – Introduces core components

• Future work

– Much more needs to be done – Performance issues – Implementation of new technologies such as WSRF

Dieter Cybok, msg systems

16

A

Acknowledgements

Most of this work was done at Imperial College London under the supervision of Dr. Steven Newhouse and Dr. Anthony Mayer