SLIDE 1
Lessons Learned Moving MAKER from HPC to the Cloud
Nick Hazekamp1, Upendra Kumar Devisetty2, Nirav Merchant2, and Douglas Thain1
1University of Notre Dame 2University of Arizona
MAKER : A Bioinformatics Pipeline
2
Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 - - PowerPoint PPT Presentation
Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 - - PowerPoint PPT Presentation
Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 , Upendra Kumar Devisetty 2 , Nirav Merchant 2 , and Douglas Thain 1 1 University of Notre Dame 2 University of Arizona MAKER : A Bioinformatics Pipeline 2 MAKER In Practice 3
SLIDE 2
SLIDE 3
MAKER In Practice
3
SLIDE 4
Jetstream Resources Available
4
SLIDE 5
MAKER Dependencies
5
SLIDE 6
VC3-Builder
6
SLIDE 7
VC3-Builder Provided Software Stack
7
SLIDE 8
MAKER in an HPC Environment
8
SLIDE 9
MPI Run on Single Node
9
SLIDE 10
Work Queue
10
SLIDE 11
WQ-MAKER
11
SLIDE 12
Work Queue Enabling Multi-instance Computation
12
SLIDE 13
Providing User Feedback - Debugging
13
SLIDE 14
Providing User Feedback - Debugging Examples
14
Level Error MAKER Invalid input format Malformed file names containing “|” char IDs too long for BLAST Work Queue Worker times out and disconnects Files fail to transfer No workers connecting VC3-Builder Environment fails to build
SLIDE 15
Providing User Feedback - Runtime
15 Sat Sep 23 15:24:25 2017 :: File Fungal annotated :: 177 Sat Sep 23 15:24:25 2017 :: Failed to annotate :: None
- Type Success Failure Abandon Total
Tasks 177 12 0 177
- Workers: Joined Removed Idled-Out Lost
10 12 2 2 0
- Work Queue Wall Time: 1d 18:02:15
Cumulative Task Wall Time: 5d 21:40:58 Cumulative Task Good Execute Time: 5d 8:33:32 Work Queue Send Time: 0d 0:11:18 Work Queue Receive Time: 0d 0:58:28
SLIDE 16
Providing User Feedback - Performance
16
Task Behavior Master Time Usage Data Transfers
SLIDE 17
Results - Fungal Genome (41MB)
17
SLIDE 18
Results - Hummingbird Genome subset (900MB)
18
SLIDE 19
Results - Cactus Genome (1.6Gb)
19
SLIDE 20
Conclusions
20
Use a tool capable of providing consistent environment on all target platform. Leverage multi-instance computation by coupling distributed communications and MPI execution. Design with feedback in mind to simplify debugging information, give runtime updates, and provide performance analysis.
SLIDE 21
21
Questions?
Nicholas Hazekamp Email : nhazekam@nd.edu
VC3-Builder CCTools MAKER
github.com/vc3-project/vc3-builder ccl.cse.nd.edu yandell-lab.org/software/maker.html