SLIDE 1
Pegasus : Introducing Integrity to Scientific Workflows
Karan Vahi
vahi@isi.edu https://pegasus.isi.edu
Pegasus
http://pegasus.isi.edu
2
Compute Compute Pipelines Pipelines – Building Building Blocks Blocks
HTCondor DAGMan
- DAGMan
DAGMan is is a a reliable reliable and and a a scalable scalable workflow workflow executor executor Sits Sits on
- n top
top of
- f HTCondor
HTCondor Schedd Schedd Can Can handle handle very very large large workflows workflows
- Has
Has useful useful reliability reliability features features in in-built built Automatic Automatic job job retries retries and and rescue rescue DAG’s DAG’s ( ( recover recover from from where where you you left left off
- ff in
in case case of
- f failures)
failures)
- Throttling
Throttling for for jobs jobs in in a a workflow workflow However, However, it it is is still still up up-to to user user to to figure figure out
- ut
- Data
Data Management Management How How do do you you ship ship in in the the small/large small/large amounts amounts data data required required by by your your pipeline pipeline and and protocols protocols to to use? use?
- How
How best best to to leverage leverage different different infrastructure infrastructure setups setups OSG OSG has has no no shared shared filesystem filesystem while while XSEDE XSEDE and and your your local local campus campus cluster cluster has has one!
- ne!
- Debug
Debug and and Monitor Monitor Computations. Computations. Correlate Correlate data data across across lots lots of
- f log
log files. files. Need Need to to know know what what host host a a job job ran ran on
- n and
and how how it it was was invoked invoked
- Restructure
Restructure Workflows Workflows for for Improved Improved Performance Performance Short hort running running tasks? tasks? Data Data placement placement