starcluster numpy scipy computing on amazon s elastic
play

StarCluster - NumPy/SciPy Computing on Amazons Elastic Compute Cloud - PowerPoint PPT Presentation

Aug 29, 2022 •186 likes •941 views

Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions StarCluster - NumPy/SciPy Computing on Amazons Elastic Compute Cloud (EC2) Justin Riley Software Tools for Academics and Researchers Office of Educational Innovation

  1. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2) Justin Riley Software Tools for Academics and Researchers Office of Educational Innovation and Technology Massachusetts Institute of Technology SciPy 2010 Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  2. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Outline 1 Introduction STAR Group Motivation behind StarCluster 2 Amazon EC2 Basics 3 StarCluster Overview Features Configuration Quick Demo Customizing StarCluster Using Elastic Block Storage Creating Plugins 4 Conclusions Future Work Where can I learn more? Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  3. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Outline 1 Introduction STAR Group Motivation behind StarCluster 2 Amazon EC2 Basics 3 StarCluster Overview Features Configuration Quick Demo Customizing StarCluster Using Elastic Block Storage Creating Plugins 4 Conclusions Future Work Where can I learn more? Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  4. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group Software Tools for Academics and Researchers 1 Work with faculty at MIT to develop software for classroom/research Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  5. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group Software Tools for Academics and Researchers 1 Work with faculty at MIT to develop software for classroom/research 2 StarBiochem - Protein Visualization Tool Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  6. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group Software Tools for Academics and Researchers 1 Work with faculty at MIT to develop software for classroom/research 2 StarBiochem - Protein Visualization Tool 3 StarGenetics - Genetic cross-simulator Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  7. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group Software Tools for Academics and Researchers 1 Work with faculty at MIT to develop software for classroom/research 2 StarBiochem - Protein Visualization Tool 3 StarGenetics - Genetic cross-simulator 4 StarMolsim - Web-based MD/Quantum simulations Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  8. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions STAR Group Software Tools for Academics and Researchers 1 Work with faculty at MIT to develop software for classroom/research 2 StarBiochem - Protein Visualization Tool 3 StarGenetics - Genetic cross-simulator 4 StarMolsim - Web-based MD/Quantum simulations 5 ... and more (http://web.mit.edu/star) Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  9. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster Motivations for StarCluster. . . Cluster Configuration is Hard Cluster Configuration Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  10. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster Motivations for StarCluster. . . Cluster Configuration is Hard Cluster Configuration Obtaining access to hardware can be a challenge Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  11. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster Motivations for StarCluster. . . Cluster Configuration is Hard Cluster Configuration Obtaining access to hardware can be a challenge Configuring and maintaing cluster configurations is hard Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  12. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster Motivations for StarCluster. . . Cluster Configuration is Hard Cluster Configuration Obtaining access to hardware can be a challenge Configuring and maintaing cluster configurations is hard Traditional resources = administrative overhead Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  13. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster StarHPC. . . SSH VNC EC2 User Virtual Desktop Figure: StarHPC Overview Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  14. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Motivation behind StarCluster StarMolSim. . . WEB SSH EC2 User MolSim using GenePattern Figure: StarMolSim Overview Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  15. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Outline 1 Introduction STAR Group Motivation behind StarCluster 2 Amazon EC2 Basics 3 StarCluster Overview Features Configuration Quick Demo Customizing StarCluster Using Elastic Block Storage Creating Plugins 4 Conclusions Future Work Where can I learn more? Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  16. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Elastic Compute Cloud Overview Infrastructure as a Service (IaaS) Cloud Computing Model Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  17. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Elastic Compute Cloud Overview Infrastructure as a Service (IaaS) Cloud Computing Model Request up to 20 virtual machines by default Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  18. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Elastic Compute Cloud Overview Infrastructure as a Service (IaaS) Cloud Computing Model Request up to 20 virtual machines by default Full root access via SSH Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  19. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Elastic Compute Cloud Overview Infrastructure as a Service (IaaS) Cloud Computing Model Request up to 20 virtual machines by default Full root access via SSH Only pay for what you use Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  20. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Elastic Block Storage Analagous to a ”Virtual USB pendrive” Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  21. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Elastic Block Storage Analagous to a ”Virtual USB pendrive” Size can be anywhere from 1GB-1TB per volume Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  22. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Elastic Block Storage Analagous to a ”Virtual USB pendrive” Size can be anywhere from 1GB-1TB per volume Supports snapshotting volumes to create backups Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  23. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Elastic Block Storage Analagous to a ”Virtual USB pendrive” Size can be anywhere from 1GB-1TB per volume Supports snapshotting volumes to create backups Ability to create new volumes based on snapshots Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

  24. Outline Introduction Amazon EC2 Basics StarCluster Overview Conclusions Standard Instances Definition 1 Compute Unit (CU) = 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. Instance Arch CPU (CU) RAM Storage I/O Cost/hr Small 32bit 1 (x1) 1.7GB 160GB Moderate $0.085 Large 64bit 2 (x2) 7.5GB 860GB High $0.34 X-Large 64bit 2 (x4) 15GB 1.69TB High $0.68 Justin Riley Massachusetts Institute of Technology StarCluster - NumPy/SciPy Computing on Amazon’s Elastic Compute Cloud (EC2)

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

SciPy: Scientic Toolkit SciPy SciPy is a collection of mathematical algorithms and convience

SciPy: Scientic Toolkit SciPy SciPy is a collection of mathematical algorithms and convience

SciPy: Scientic Toolkit SciPy SciPy is a collection of mathematical algorithms and convience functions built on Numpy data structures Organized into subpackages covering dierent scientic computing areas A data-processing and

526 views • 7 slides
Instance Support Elastic Load Balancing Amazon EC2 AWS Elastic Beanstalk Amazon EC2 Container

Instance Support Elastic Load Balancing Amazon EC2 AWS Elastic Beanstalk Amazon EC2 Container

Instance Support Elastic Load Balancing Amazon EC2 AWS Elastic Beanstalk Amazon EC2 Container Amazon ECS VMWare Cloud on AWS Registry Amazon Lightsail AWS Outposts AWS Batch Container Functions (Serverless Compute) Amazon ECS* AWS

347 views • 15 slides
An Introduction to Numpy Thomas Schwarz, SJ NumPy Fundamentals Numpy is a module for faster

An Introduction to Numpy Thomas Schwarz, SJ NumPy Fundamentals Numpy is a module for faster

An Introduction to Numpy Thomas Schwarz, SJ NumPy Fundamentals Numpy is a module for faster vector processing with numerous other routines Scipy is a more extensive module that also includes many other functionalities such as machine

378 views • 35 slides
Lin inear programming Example Numpy: PageRank scipy.optimize.linprog Example linear

Lin inear programming Example Numpy: PageRank scipy.optimize.linprog Example linear

Lin inear programming Example Numpy: PageRank scipy.optimize.linprog Example linear programming: Maximum flow PageRank PageRank - A A NumPy / / Jupyter / / matplotlib example Central to Google's original search engine was the

461 views • 22 slides
iPython Data Analytics in Python 1 / 13 The SciPy Stack SciPy is a Python-based ecosystem of

iPython Data Analytics in Python 1 / 13 The SciPy Stack SciPy is a Python-based ecosystem of

iPython Data Analytics in Python 1 / 13 The SciPy Stack SciPy is a Python-based ecosystem of libraries and tools for scientific computing and data analytics iPython Jupyter notebooks Numpy Pandas Matplotlib iPython is the

608 views • 13 slides
Reducing Costs of Spot Instances via Checkpointing in the Amazon Elastic Compute Cloud - Qingxi

Reducing Costs of Spot Instances via Checkpointing in the Amazon Elastic Compute Cloud - Qingxi

Reducing Costs of Spot Instances via Checkpointing in the Amazon Elastic Compute Cloud - Qingxi Li 1 Outline Amazon Elastic Compute Cloud Checkpointing 2 Cloud Computing Cloud computing is a model for enabling convenient, on-

832 views • 56 slides
VMD & NAMD on Elastic Compute Cloud (EC2) instance of Amazon Web Services (AWS) Start VMD

VMD & NAMD on Elastic Compute Cloud (EC2) instance of Amazon Web Services (AWS) Start VMD

VMD & NAMD on Elastic Compute Cloud (EC2) instance of Amazon Web Services (AWS) Start VMD & NAMD AMI (once you have created your AWS account) AMI - Amazon Machine Image Amazon Marketplace - https://aws.amazon.com/marketplace/ Search for

709 views • 36 slides
Connecting ROOT to the Python world with Numpy arrays 2018-03-08 1 What is the idea? Numpy

Connecting ROOT to the Python world with Numpy arrays 2018-03-08 1 What is the idea? Numpy

Connecting ROOT to the Python world with Numpy arrays 2018-03-08 1 What is the idea? Numpy arrays are the interface for all of the scientific libraries in the Python world (scipy, sklearn, tensorflow, matplotlib, . . . ). The desired

750 views • 5 slides
Amazon Elastic Compute Cloud (EC2) vs. in-House HPC Platform a Cost Analysis J. Emeras, S.

Amazon Elastic Compute Cloud (EC2) vs. in-House HPC Platform a Cost Analysis J. Emeras, S.

Amazon Elastic Compute Cloud (EC2) vs. in-House HPC Platform a Cost Analysis J. Emeras, S. Varrette, P. Bouvry Parallel Computing and Optimization Group (PCOG), University of Luxembourg (UL), Luxembourg S. Varrette (PCOG Research unit)

670 views • 38 slides
AMath 483/583 Lecture 6 Notes: This lecture: NumPy arrays and functions Python: main

AMath 483/583 Lecture 6 Notes: This lecture: NumPy arrays and functions Python: main

AMath 483/583 Lecture 6 Notes: This lecture: NumPy arrays and functions Python: main programs and private variables Timing Python execution Reading: class notes: Numerical Python Numpy and Scipy documentation R.J.

383 views • 6 slides
http://data-mining-tutorials.blogspot.fr/ 1 R.R. Universit Lyon 2 Numpy ? NumPy

http://data-mining-tutorials.blogspot.fr/ 1 R.R. Universit Lyon 2 Numpy ? NumPy

Ricco Rakotomalala http://data-mining-tutorials.blogspot.fr/ 1 R.R. Universit Lyon 2 Numpy ? NumPy (numerical python) is a package for scientific computing. It provides tools for handling n-dimensional arrays (especially vectors and

358 views • 19 slides
Scientific Programming with the SciPy Stack Shaun Walbridge Kevin Butler

Scientific Programming with the SciPy Stack Shaun Walbridge Kevin Butler

Scientific Programming with the SciPy Stack Shaun Walbridge Kevin Butler https://github.com/scw/scipy-devsummit-2016-talk Handout PDF High Quality PDF (5MB) Resources Section Scientific Computing Scientific Computing The application of

661 views • 19 slides
Numpy: Vectorize your brain K nearest neighbors https://archive.ics.uci.edu/ml/datasets/Wine

Numpy: Vectorize your brain K nearest neighbors https://archive.ics.uci.edu/ml/datasets/Wine

Ekaterina Tuzova Numpy: Vectorize your brain K nearest neighbors https://archive.ics.uci.edu/ml/datasets/Wine NumPy What is NumPy? Numpy is the fundamental package for scientific computing with Python. IPython Python and Performance Python

733 views • 57 slides
Numerical Calculations Ali Taheri Sharif University of Technology Fall 2018 Some slides have

Numerical Calculations Ali Taheri Sharif University of Technology Fall 2018 Some slides have

Fu Fundamentals of Pr Programming (Py Python) Numerical Calculations Ali Taheri Sharif University of Technology Fall 2018 Some slides have been adapted from Scipy Lecture Notes at http://www.scipy-lectures.org/ Outline 1. NumPy and

625 views • 36 slides
A Short History of Array Computing in Python Wolf Vollprecht, PyParis 2018 TOC - Array

A Short History of Array Computing in Python Wolf Vollprecht, PyParis 2018 TOC - Array

A Short History of Array Computing in Python Wolf Vollprecht, PyParis 2018 TOC - Array computing in general - History up to NumPy - Libraries after NumPy - Pure Python libraries - JIT / AOT compilers - Deep Learning - NumPy

585 views • 45 slides
Amazon EC2, GPU computing, PyNX:Ptychography Vincent Favre-Nicolin X-ray NanoProbe group, ESRF 1

Amazon EC2, GPU computing, PyNX:Ptychography Vincent Favre-Nicolin X-ray NanoProbe group, ESRF 1

Amazon EC2, GPU computing, PyNX:Ptychography Vincent Favre-Nicolin X-ray NanoProbe group, ESRF 1 l 66TH MEETING OF THE ESRF l 30-31 May 2014 l Author 26/07/2013 AMAZON GPU COMPUTING Gpu Accelerated computing instances: (old) G2: nVidia

73 views • 6 slides
Library2Go for Kindle - Instructions Kindle Fire, Smartphone and Tablet users: You must have

Library2Go for Kindle - Instructions Kindle Fire, Smartphone and Tablet users: You must have

Library2Go for Kindle - Instructions Kindle Fire, Smartphone and Tablet users: You must have access to the Library2Go full site (not the mobile site) to access Kindle eBooks directly from your device. Click the link on the E-book help page

384 views • 24 slides
Scalable Gaussian Processes Zhenwen Dai Amazon 9 September 2019 @GPSS 2019 Zhenwen Dai (Amazon)

Scalable Gaussian Processes Zhenwen Dai Amazon 9 September 2019 @GPSS 2019 Zhenwen Dai (Amazon)

Scalable Gaussian Processes Zhenwen Dai Amazon 9 September 2019 @GPSS 2019 Zhenwen Dai (Amazon) Scalable Gaussian Processes 9 September 2019 @GPSS 2019 1 / 46 Gaussian process Input and Output Data: X = ( x 1 , . . . , x N ) y = ( y 1 ,

516 views • 47 slides
Securing Serverless and Container Services Marc Schrter AWS DevOps Engineer @ globaldatanet

Securing Serverless and Container Services Marc Schrter AWS DevOps Engineer @ globaldatanet

Securing Serverless and Container Services Marc Schrter AWS DevOps Engineer @ globaldatanet Community Day 2019 Sponsors Serverless DevOps Automation Continuous Delivery Highly scalable and Infrastructure as Code fault-tolerant solutions

410 views • 40 slides
Internet censorship is everywhere Source:

Internet censorship is everywhere Source:

Internet censorship is everywhere Source: https://freedomhouse.org/report/freedom-net/freedom-net-2016 Commonly censored content Unapproved news (Fake News!!111) Wikipedia (usually partially) Facebook Google (all services), YouTube

539 views • 37 slides
Basheer Al-Duwairi Jordan University of Science & Technology United-States/Middle-East

Basheer Al-Duwairi Jordan University of Science & Technology United-States/Middle-East

Basheer Al-Duwairi Jordan University of Science & Technology United-States/Middle-East Workshop On Trustworthiness in Emerging Distributed Systems and Networks, Istanbul- Turkey, 4-6 June 2012 Outline Examples of using network

301 views • 12 slides
Video Codecs In An AI World Dr Doug Ridge Amphion Semiconductor The Proliferance of Video in

Video Codecs In An AI World Dr Doug Ridge Amphion Semiconductor The Proliferance of Video in

Video Codecs In An AI World Dr Doug Ridge Amphion Semiconductor The Proliferance of Video in Networks Video produces huge volumes of data According to Cisco By 2021 video will make up 82% of network traffic Equals 3.3

429 views • 13 slides
A"Large(Scale"Analysis"of"the"

A"Large(Scale"Analysis"of"the"

A"Large(Scale"Analysis"of"the" Security"of"Embedded"Firmwares Presented(by(Zhenyu Ning 1 Contents 1.(Background 2.(Motivation(&(Challenges 3.(Architecture 4.(Analysis(Result(&(Case(study

784 views • 23 slides
A Probabilistic U-Net for Segmentation of Ambiguous Images Simon A. A. Kohl 1*,2 , Bernardino

A Probabilistic U-Net for Segmentation of Ambiguous Images Simon A. A. Kohl 1*,2 , Bernardino

A Probabilistic U-Net for Segmentation of Ambiguous Images Simon A. A. Kohl 1*,2 , Bernardino Romera-Paredes 1 , Clemens Meyer 1 , Jeffrey De Fauw 1 , Joseph R. Ledsam 1 , Klaus H. Maier-Hein 2 , S. M. Ali Eslami 1 , Danilo Jimenez Rezende 1 , Olaf

669 views • 32 slides

More recommend