Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 - PowerPoint PPT Presentation

Oct 15, 2022 •28 likes •238 views

Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 , Upendra Kumar Devisetty 2 , Nirav Merchant 2 , and Douglas Thain 1 1 University of Notre Dame 2 University of Arizona MAKER : A Bioinformatics Pipeline 2 MAKER In Practice 3

Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 , Upendra Kumar Devisetty 2 , Nirav Merchant 2 , and Douglas Thain 1 1 University of Notre Dame 2 University of Arizona
MAKER : A Bioinformatics Pipeline 2
MAKER In Practice 3
Jetstream Resources Available 4
MAKER Dependencies 5
VC3-Builder 6
VC3-Builder Provided Software Stack 7
MAKER in an HPC Environment 8
MPI Run on Single Node 9
Work Queue 10
WQ-MAKER 11
Work Queue Enabling Multi-instance Computation 12
Providing User Feedback - Debugging 13
Providing User Feedback - Debugging Examples Level Error MAKER Invalid input format Malformed file names containing “|” char IDs too long for BLAST Work Queue Worker times out and disconnects Files fail to transfer No workers connecting VC3-Builder Environment fails to build 14
Providing User Feedback - Runtime Sat Sep 23 15:24:25 2017 :: File Fungal annotated :: 177 Sat Sep 23 15:24:25 2017 :: Failed to annotate :: None ----------------------------------------------------------------- Type Success Failure Abandon Total Tasks 177 12 0 177 ----------------------------------------------------------------- Workers: Joined Removed Idled-Out Lost 10 12 2 2 0 ----------------------------------------------------------------- Work Queue Wall Time: 1d 18:02:15 Cumulative Task Wall Time: 5d 21:40:58 Cumulative Task Good Execute Time: 5d 8:33:32 Work Queue Send Time: 0d 0:11:18 Work Queue Receive Time: 0d 0:58:28 ----------------------------------------------------------------- 15
Providing User Feedback - Performance Task Behavior Master Time Usage Data Transfers 16
Results - Fungal Genome (41MB) 17
Results - Hummingbird Genome subset (900MB) 18
Results - Cactus Genome (1.6Gb) 19
Conclusions Use a tool capable of providing consistent environment on all target platform. Leverage multi-instance computation by coupling distributed communications and MPI execution. Design with feedback in mind to simplify debugging information, give runtime updates, and provide performance analysis. 20
Questions? VC3-Builder CCTools MAKER github.com/vc3-project/vc3-builder ccl.cse.nd.edu yandell-lab.org/software/maker.html Nicholas Hazekamp 21 Email : nhazekam@nd.edu

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Nico Uys Cloud Business Line Manager 1 Recent SAP on cloud projects Lessons learned

SAP in the cloud, do it right. Nico Uys Cloud Business Line Manager 1 Recent SAP on cloud projects Lessons learned Agenda Automation of the cloud Zag and our services Summary Q & A 2 Interesting Cloud Facts 1.

271 views • 23 slides

Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From

Protg 2006, July 26th, Stanford I R I S A C 1 R E C Saint-Cyr Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From Ontology Ontology Design Design Ontology Ontology Design Design Jean Andr B

371 views • 12 slides

Lessons Learned Automating Cloud and Infrastructure Testing

T23 DevOps & The Cloud 2019-05-02 15:00 Lessons Learned Automating Cloud and Infrastructure Testing Presented by:

729 views • 40 slides

Lessons learned from deploying SUSE OpenStack Cloud and Enterprise Storage in the Public Cloud

Lessons learned from deploying SUSE OpenStack Cloud and Enterprise Storage in the Public Cloud TUT1224 Thursday, April 04, 03:15 PM - 04:15 PM | Belmont 1 Friday, April 05, 10:15 AM - 11:15 AM | Belmont 2 Mike Friesenegger Solution Architect

683 views • 35 slides

Security in a cloud context David Crooks, for the EGI CSIRT Lessons learned from recent incidents

Security in a cloud context David Crooks, for the EGI CSIRT Lessons learned from recent incidents www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142 Cloud Security EGI

571 views • 16 slides

C++, Java and .NET: Lessons learned from the Internet Age, and What it Means for the Cloud and

<Insert Picture Here> C++, Java and .NET: Lessons learned from the Internet Age, and What it Means for the Cloud and Emerging Languages Cameron Purdy Vice President, Development Overview <Insert Picture Here> A retrospective of

532 views • 40 slides

Moving Forward: Lessons Learned From The Last 10 Years of Risk Modeling Presentation Agenda 1)

Moving Forward: Lessons Learned From The Last 10 Years of Risk Modeling Presentation Agenda 1) Provide a framework for defining risk 2) Examine the role of theory and personal bias in the modeling process 3) Help outline possible partnerships

558 views • 27 slides

Cloud Computing Lessons Learned & Technical Perspective michal.furmankiewicz@chmurowisko.pl

Cloud Computing Lessons Learned & Technical Perspective michal.furmankiewicz@chmurowisko.pl https://www.linkedin.com/in/mifurm/ @mifurm Micha Furmankiewicz Cloud Architect for last 7 years CEO & co-owner of Chmurowisko / Cloud

834 views • 35 slides

Applying Architectural Patterns for the Cloud: Lessons Learned During Pattern Mining and

Applying Architectural Patterns for the Cloud: Lessons Learned During Pattern Mining and Application Ralph Retter (Daimler TSS GmbH) ralph.retter@daimler.com Christoph Fehling (University of Stuttgart, Germany) Problem We need to do Cloud

282 views • 25 slides

Lessons learned from six years of cloud technology transformation in government David Turner

Lessons learned from six years of cloud technology transformation in government David Turner Managing Director, MSC Digital MSC Digital Independent and vendor agnostic Formed specifically to assist the UK Public Sector - Its

386 views • 19 slides

Data Acquisition Methods: Lessons Learned from Deployment of Cloud HVAC Analytics at UCSB

Data Acquisition Methods: Lessons Learned from Deployment of Cloud HVAC Analytics at UCSB Tuesday, June 27 th 2017 Michael Georgescu (Ecorithm) Jordan Sager (UCSB) California Higher Education & Sustainability Conference (CHESC 2017)

452 views • 16 slides

The 'Cloud Area Padovana': lessons learned a3er two years of a produc:on OpenStack-based IaaS for

The 'Cloud Area Padovana': lessons learned a3er two years of a produc:on OpenStack-based IaaS for the local INFN user community Interna:onal Symposium on Grids and Clouds (ISGC) 2017 Academia Sinica, Taipei, Taiwan, 5-10 March 2017 Marco

295 views • 25 slides

Architecting for the cloud: lessons learned from 100 CloudStack deployments Sheng Liang CTO,

Architecting for the cloud: lessons learned from 100 CloudStack deployments Sheng Liang CTO, Cloud Platforms, Citrix CloudStack History 2008 2009 2010 2012 2011 Sept 2008: Nov 2009: May 2010: July 2011: April 2012: CloudStack

457 views • 34 slides

1 <Insert Picture Here> C++, Java and .NET: Lessons learned from the Internet Age, and

1 <Insert Picture Here> C++, Java and .NET: Lessons learned from the Internet Age, and What it Means for the Cloud and Emerging Languages Cameron Purdy Vice President, Development Overview <Insert Picture Here> A retrospective of

656 views • 41 slides

1. Building Description and Background 2. Fire 3. Inquest 4. Challenges 5. Lessons Learned 6.

917 Dundas Street W, Whitby 2016 OMFPOA Symposium Fire Chief Dave Speed CFPO Nicholas Webb 1. Building Description and Background 2. Fire 3. Inquest 4. Challenges 5. Lessons Learned 6. Moving Forward 1) Building Description and Background

468 views • 15 slides

Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background

Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background SEP 2002 Davis-Besse Lessons Learned Task Force AUG 2004 - Effectiveness Review Lessons Learned Task Force JAN 2005 - EDO Charters

269 views • 22 slides

Elect ectric ic Machin ine e Design ign Stef efan n Holst, lst, CD CD-adapco dapco

The use of Simulation mulation in Elect ectric ic Machin ine e Design ign Stef efan n Holst, lst, CD CD-adapco dapco Motiv ivati ation on How w often n can a machine hine be started ed with thin n an hour ur In Hybrids

230 views • 18 slides

EVENTSTORMING COLLABORATIVE LEARNING FOR COMPLEX DOMAINS Paul Rayner @thepaulrayner THE

EVENTSTORMING COLLABORATIVE LEARNING FOR COMPLEX DOMAINS Paul Rayner @thepaulrayner THE LANDSCAPE Complex business rules & processes Knowledge silos Numerous diverse (& often legacy) system interactions THE CRITICAL COMPLEXITY OF

1.13k views • 80 slides

RAD-seq in Roscoff Matthieu Bruneaux 2015-03-10 Mini-workshop about ddRAD Introduction about

RAD-seq in Roscoff Matthieu Bruneaux 2015-03-10 Mini-workshop about ddRAD Introduction about RAD-seq RAD? RAD-seq? ddRAD? Applications Workflow Practicals One complete project, from raw reads to final results Cherry-picking

686 views • 45 slides

Use of Numerical Modelling to Mitigate Ground Risk CECA MEETING SEPT 2017 Gavin & Doherty

Use of Numerical Modelling to Mitigate Ground Risk CECA MEETING SEPT 2017 Gavin & Doherty Geosolutions Ltd. Overview Todays Presentation. o Introduction to GDG o Finite Element Modelling & Calibration o Case Studies o Flood

788 views • 64 slides

Identification of haplotypes controlling seedless by genome resequencing of grape Soon-Chun

Identification of haplotypes controlling seedless by genome resequencing of grape Soon-Chun Jeong scjeong@kribb.re.kr Korea Research Institute of Bioscience and Biotechnology Why seedless grape research in Korea? Among the three major uses

342 views • 12 slides

Junguk Hur University of North Dakota School of Medicine and Health Sciences hurlab.med.und.edu

Integration of machine learning- and dictionary-based approach for identification of adverse drug reactions in drug labels Junguk Hur University of North Dakota School of Medicine and Health Sciences hurlab.med.und.edu Team: CONDL C

432 views • 21 slides

Som e Aspects of Vision 2 0 3 0 Conference On Security And Cooperation In South Asia: A Global

Som e Aspects of Vision 2 0 3 0 Conference On Security And Cooperation In South Asia: A Global Perspective October 8-10, 2007 Berlin, Germany Dr Shaukat Hameed Khan Planning Commission Government of Pakistan Islamabad Vision 2030

616 views • 38 slides

California Workers Compensation Reform What Should We Expect Next Dave Bellusci, FCAS, MAAA

California Workers Compensation Reform: What Should We Expect Next California Workers Compensation Reform What Should We Expect Next Dave Bellusci, FCAS, MAAA WCIRB Michele Lyons, RN, MSN, JD, CCM, MSCC Farmers Insurance Casualty Loss

649 views • 54 slides