Apache Airavata Open Community Science Gateway Framework Shahbaz Memon * , Suresh Marru + , Marlon Pierce + * Juelich Supercomputing Centre + University of Indiana Bloomington
Outline • Introduce Apache Airavata • High level architecture • Generic Science Gateway API • Motivation for Apache -- Governance Model • Ultrascan as a use case
What Is Apache Airavata? • Science Gateway software system to • Compose, manage, execute, and monitor distributed, computational workflows • Wrap legacy command line scientific applications with Web services. • Run jobs on computational resources ranging from local resources to computational grids and clouds • Airavata software is largely evolved from NSF-funded academic research.
Airavata Legacy into Apache Open Gateway EHAT/MEAD/LEAD Computing Apache Airavata Gateways Environments (2011 to …) (1999-2009) (2006 to ….)
Apache Airavata http://airavata.apache.org
Apache Airavata L o r i e n m s d o i u p m l p x o 1 e 5 s n u s m Core End Users Developer Message Box Scientific Applicati on Apache Airavata API Application Gateway Developer Workflow Factory Interpreter Computational Resources Registry
Connecting a Science Gateway to Airavata Computational Resources
Apache Airavata Components Component Description XBaya Workflow graphical composition tool. Registry Service Insert and access application, host machine, workflow, and provenance data. Workflow Interpreter Execute the workflow on one or more resources. Service Application Factory Manages the execution and management of an Service (GFAC) application in a workflow Message Box WS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events Airavata API Single wrapping client to provide higher level programming interfaces.
Mapping Science to Grids Define Experiment Define Workflow Setup Input Setup Context Update exec. metadata Configure Experiment Setup Persist Forms Data Run Experiment Run Workflow Retrieve Provenance Check Progress Airavata Server Monitor Execution Progres s Calculat Airavata API Setup notifications e Security Computing Stateful Monitor Execution Listener Resources Retrieve Provenance Analyze Results Retrieve Errors Visualiz e Results Scientist Analyze Errors Retrieve Errors Translat Verify access e Errors Update ACL Share Results Replicate data Setup Notify ACL Peers Retrieve job details Repeat/Retry Configure new execution Cancel Identify Cancel current execution & Retry Job Redo Workflow/node
Interoperable Workflow Architecture
Application Factory Global In Global In Global In Application Provider Global In Global In Global In Global Handlers Handlers Handlers Specific Specific Handlers Handlers Handlers InHandlers InHandlers InHandlers Provider Logic Global In Global In Global In Application Provider Global In Global In Global In Global Handlers Handlers Handlers Specific Specific Handlers Handlers Handlers OutHandlers OutHandlers OutHandlers
Open Community Software and Governance • Open source projects need diversity, governance. – Reproducibility – Sustainability Compete • Incentives for projects to diversify their developer base. • Govern • Software releases • Contributions • Credit sharing. • Members are added • Project direction decisions. Collaborate • IP, legal issues • Our approach: Apache Software Foundation
Apache Way: Beyond Open Source Communities • Transparency – Decision-making and actions are observable – Events of interest are published and recorded – Transparency invites collaboration • Meritocratic Governance – Influence on decisions is based on merit – Merit is earned in public – Community based governance • Community – Common interest, Community interest, Common experience – “Community before code” • Collaboration – Systems supporting communication and coordination: repositories, trackers, forums, build tools – You can reuse what you can see and influence – More eyeballs means better quality
Apache Airavata in Action Domain Description Astronomy Image processing pipeline for One Degree Imager instrument on XSEDE Astrophysics Supporting workflow of Dark Energy Survey simulations working group on XSEDE Bioinformatics Supported workflow executions on Amazon EC2 for BioVLAB project Biophysics Manage large scale data analysis of analytical ultracentrifugation experiments on XSEDE and campus resources Computational Manage workflows to support computational Chemistry chemistry parameter studies for ParamChem.org on XSEDE Nuclear Physics Workflows for nuclear structure calculations using Leadership Class Configuration Interaction (LCCI) computations on DOE resources
The Scientific Case: Ultrascan 3 (US3) US3 interprets complex data coming from high resolution analytical ultracentrifugation (AUC) experiments using sedimentation velocity centrifugation methods It provides better understanding of biological macromolecules and synthetic polymers that allows in the investigation of structural relationships of biological systems in many diseases such as different kind of cancers Users can fit their experiments using a 2-dimensional spectrum analysis, genetic algorithm optimization and Monte Carlo analysis to obtain fine grained details and analyze noise components in the data 17. September 2013 16
US3 LIMS – Apache Airavata – Middlewares US-LIMS UNICORE Client API JUROPA 17
UltraScan Science Gateway Use Case A 1 B 2 UltraScan LIMS 3 Apache Airavata OGSA-BES/JSDL Native Native XSEDE Juelich
Conclusions • Apache Airavata is a general purpose software for science gateways • The open community model of Apache foundation encourages a neutral venue for collaborating on Science Gateways Foundational Layers. • Adding a UNICORE provider to Airavata was a seamless experience. Future work includes developing more support on XSEDE and PRACE. • Airavata follows a application/use case driven development and has played less attention to standards. – This is an area which can benefit from future collaboration with XSEDE/PRACE and other interoperability groups.
Questions?
Recommend
More recommend
