Apache Airavata Open Community Science Gateway Framework Shahbaz - - PowerPoint PPT Presentation

Apache Airavata

Open Community Science Gateway Framework

Shahbaz Memon*, Suresh Marru+, Marlon Pierce+

*Juelich Supercomputing Centre +University of Indiana Bloomington

Outline

• Introduce Apache Airavata
• High level architecture
• Generic Science Gateway API
• Motivation for Apache -- Governance Model
• Ultrascan as a use case

What Is Apache Airavata?

• Science Gateway software system

to

• Compose, manage, execute, and

monitor distributed, computational workflows

• Wrap legacy command line scientific

applications with Web services.

• Run jobs on computational

resources ranging from local resources to computational grids and clouds

• Airavata software is largely

evolved from NSF-funded academic research.

Airavata Legacy into Apache

EHAT/MEAD/LEAD Gateways (1999-2009) Open Gateway Computing Environments (2006 to ….) Apache Airavata (2011 to …)

Apache Airavata

http://airavata.apache.org

Workflow Interpreter Application Factory Message Box Registry

Apache Airavata API

L

• r

e m i p s u m i n s

• l

e n s p 1 m 5 d u

• x

End Users Gateway Developer

Scientific Applicati

• n

Core Developer

Computational Resources

Apache Airavata

Connecting a Science Gateway to Airavata

Computational Resources

Apache Airavata Components

Component Description XBaya Workflow graphical composition tool. Registry Service Insert and access application, host machine, workflow, and provenance data. Workflow Interpreter Service Execute the workflow on one or more resources. Application Factory Service (GFAC) Manages the execution and management of an application in a workflow Message Box WS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events Airavata API Single wrapping client to provide higher level programming interfaces.

Mapping Science to Grids

Airavata Server

Security

Define Experiment Configure Experiment Run Experiment Check Progress Analyze Errors Analyze Results Share Results Setup notifications Repeat/Retry

Define Workflow Setup Input Setup Context Update exec. metadata Run Workflow Retrieve Provenance Monitor Execution Monitor Execution Retrieve Provenance Retrieve Errors Retrieve Errors Verify access Update ACL Replicate data Retrieve job details Configure new execution Cancel current execution Redo Workflow/node

Computing Resources

Scientist

Airavata API

Stateful Listener Progres s Calculat e Setup Forms Persist Data Visualiz e Results Translat e Errors Setup ACL Notify Peers Identify Job Cancel & Retry

Interoperable Workflow Architecture

Global In Handlers Global In Handlers Global InHandlers Global In Handlers Global In Handlers Application Specific InHandlers Global In Handlers Global In Handlers Provider Specific InHandlers Global In Handlers Global In Handlers Global OutHandlers Global In Handlers Global In Handlers Application Specific OutHandlers Global In Handlers Global In Handlers Provider Specific OutHandlers Provider Logic

Application Factory

Open Community Software and Governance

• Open source projects need

diversity, governance.

– Reproducibility – Sustainability

• Incentives for projects to

diversify their developer base.

• Govern
• Software releases
• Contributions
• Credit sharing.
• Members are added
• Project direction

decisions.

• IP, legal issues
• Our approach: Apache

Software Foundation

Collaborate Compete

Apache Way: Beyond Open Source Communities

• Transparency

– Decision-making and actions are observable – Events of interest are published and recorded – Transparency invites collaboration

• Meritocratic Governance

– Influence on decisions is based on merit – Merit is earned in public – Community based governance

• Community

– Common interest, Community interest, Common experience – “Community before code”

• Collaboration

– Systems supporting communication and coordination: repositories, trackers, forums, build tools – You can reuse what you can see and influence – More eyeballs means better quality

Domain Description Astronomy Image processing pipeline for One Degree Imager instrument on XSEDE Astrophysics Supporting workflow of Dark Energy Survey simulations working group on XSEDE Bioinformatics Supported workflow executions on Amazon EC2 for BioVLAB project Biophysics Manage large scale data analysis of analytical ultracentrifugation experiments on XSEDE and campus resources Computational Chemistry Manage workflows to support computational chemistry parameter studies for ParamChem.org on XSEDE Nuclear Physics Workflows for nuclear structure calculations using Leadership Class Configuration Interaction (LCCI) computations on DOE resources

Apache Airavata in Action

• US3 interprets complex data coming from high resolution analytical

ultracentrifugation (AUC) experiments using sedimentation velocity centrifugation methods

• It provides better understanding of biological macromolecules and

synthetic polymers that allows in the investigation of structural relationships of biological systems in many diseases such as different kind of cancers

• Users can fit their experiments using a 2-dimensional spectrum

analysis, genetic algorithm optimization and Monte Carlo analysis to

• btain fine grained details and analyze noise components in the data
• 17. September 2013

16

The Scientific Case: Ultrascan 3 (US3)

US3 LIMS – Apache Airavata – Middlewares

UNICORE Client API

JUROPA

US-LIMS

17

UltraScan Science Gateway Use Case

UltraScan LIMS Apache Airavata

A B 1 2 3

XSEDE Juelich

OGSA-BES/JSDL Native Native

Conclusions

• Apache Airavata is a general purpose software for science

gateways

• The open community model of Apache foundation

encourages a neutral venue for collaborating on Science Gateways Foundational Layers.

• Adding a UNICORE provider to Airavata was a seamless
• experience. Future work includes developing more support on

XSEDE and PRACE.

• Airavata follows a application/use case driven development

and has played less attention to standards.

– This is an area which can benefit from future collaboration with XSEDE/PRACE and other interoperability groups.

Questions?