beyond data and model parallelism for deep neural networks
play

Beyond Data and Model Parallelism for Deep Neural Networks ZHIHAO - PowerPoint PPT Presentation

Sep 08, 2023 •163 likes •284 views

Beyond Data and Model Parallelism for Deep Neural Networks ZHIHAO JIA, MATEI ZAHARIA, ALEX AIKEN SYSML 2019 PRESENTED BY JULIUS LISCHEID Existing Parallelisation Approaches (1/2) DATA PARALLELISM MODEL PARALLELISM Replica of neural

  1. Beyond Data and Model Parallelism for Deep Neural Networks ZHIHAO JIA, MATEI ZAHARIA, ALEX AIKEN SYSML 2019 PRESENTED BY JULIUS LISCHEID

  2. Existing Parallelisation Approaches (1/2) DATA PARALLELISM MODEL PARALLELISM • Replica of neural network on each device • Disjoint subsets of neural network assigned to devices • Each device processes subset of training data • No parameter synchronisation, but requires • After each iteration, parameters are data transfers between operations synchronised • Works well for compute-heavy operations with few parameters (e.g. convolutions)

  3. Existing Parallelisation Approaches (2/2) EXPERT-DESIGNED STRATEGIES AUTOMATED FRAMEWORKS • A. Krizhevsky. One weird trick for parallelizing • A. Mirhoseini et al. Device Placement convolutional neural networks. CoRR 2014. Optimization with Reinforcement Learning. ICML 2017. • Data parallelism for convolutional layers, model • Reinforment learning for model parallelism parallelism for fully-connected layers • Y. Wu et al. Google’s neural machine translation • Z. Jia et al. Exploring hidden dimensions in system: bridging the gap between human and parallelizing convolutional neural networks. CoRR machine translation. CoRR 2016. 2018. • Data parallelism for compute nodes, model • Dynamic Programming for parallelisation of DNNs parallelism for intra-node computation with linear computation graphs • D. Narayanan et al. PipeDream: generalized pipeline parallelism for DNN training. SOSP 2019. • …

  4. The SOAP Search Space Samples (data parallelism) Operators (model parallelism) Attributes (e.g. pixels) Parameters (≈model parallelism)

  5. Hybrid Parallelism in SOAP Example parallelization strategies for 1D convolution

  6. FlexFlow • Trying out strategies on hardware is expensive due to long iteration times • Execution Optimizer uses simulator instead • Measures operator runtime on hardware • Estimates runtime of parallelisation strategies • Delta simulation algorithm uses incremental updates for acceleration • Execution optimizer explores search space with Markov Chain Monte Carlo algorithm

  7. Evaluation (1/2)

  8. Evaluation (2/2)

  9. Review (1/2) STRENGTHS/AGREEMENTS WEAKNESSES/DISAGREEMENTS • Expands search space for parallelisation • Unclear how much SOAP and execution strategies optimiser contribute to training acceleration • Proposes a way to efficiently explore that • Usefulness of Attribute dimension is search space questionable • Leads to an actual speed-up • More end-to-end performance benchmarks would have been useful

  10. Review (2/2) KEY TAKEAWAYS POTENTIAL IMPACT • Training performance of parallelisation • Usage of other search algorithms to explore strategies can be efficiently and accurately parallelisation search space in simulation predicted • Combination of parallelisation search space • The resulting speed-up allows for the with computation graph substitutions exploration of a wider search space (compare Tim’s presentation next week)

  11. Questions? P A S O

  12. Image Citations Images with beige background retrieved from Jia Zhihao’s SysML 19 talk: https://www.youtube.com/watch?v=81l6kkV-OkE All other images extracted from Z. Jia, M. Zaharia, and A. Aiken: Beyond Data and Model Parallelism for Deep Neural Networks, SYSML, 2019.

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Beyond Data and Model Parallelism for Deep Neural Networks Zhihao Jia, Matei Zaharia and Alex

Beyond Data and Model Parallelism for Deep Neural Networks Zhihao Jia, Matei Zaharia and Alex

Beyond Data and Model Parallelism for Deep Neural Networks Zhihao Jia, Matei Zaharia and Alex Aiken Cristian (cb2015@cam.ac.uk) Content Types of parallelism The SOAP space FlexFlow Evaluation of FlexFlow Critique Types of

420 views • 18 slides
Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2 1 University of Oxford, 2 EPFL ICLR 2020 Workshop on PML4DC Motivation Compressing neural networks can save a large amount of memory and

570 views • 21 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep

AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep

AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep Ll. Berral @BSC Intel Academic Education Mindshare Initiative for AI Introduction Neural networks attempt to imitate the brain neuron

339 views • 21 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long

Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long

Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long Short-Term Memory: LSTM

366 views • 18 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.37k views • 134 slides
WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1.

WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1.

OPTICAL FLOW ESTIMATION WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1. Optical Flow Motivation 2. Neural Networks Basics 3. Optical Flow with Deep Neural Networks 1. PWC-Net Model 2. PWC-Net Results

481 views • 20 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Learning from Limited Data The University of Tokyo / RIKEN AIP Tatsuya Harada Deep Neural

Learning from Limited Data The University of Tokyo / RIKEN AIP Tatsuya Harada Deep Neural

GTC March 18, 2019 Learning from Limited Data The University of Tokyo / RIKEN AIP Tatsuya Harada Deep Neural Networks for Visual Recognition Deep Neural Networks Applications A yellow train on the tracks near a train station. cellphone cup

661 views • 53 slides
dhcd Department of Housing and Community Development Department of Housing and Community

dhcd Department of Housing and Community Development Department of Housing and Community

dhcd Department of Housing and Community Development Department of Housing and Community Development Pre-Solicitation and Capacity Building Workshop February 27, 2014 WELCOME 1 dhcd Department of Housing and Community Development Department

816 views • 50 slides
1 Business & Operating Highlights Overview of the Bank 03 Review of Operating Environment

1 Business & Operating Highlights Overview of the Bank 03 Review of Operating Environment

1 Business & Operating Highlights Overview of the Bank 03 Review of Operating Environment 04 Policy Highlights 05 Recap of 2018 Commitments 06 ALAT Full Digital Bank 07 Leadership Transition 08 1 2 P r e s e n t a t i o n t

816 views • 28 slides
BIOLOGICAL REMOVAL OF HYDROGEN SULPHIDE FROM LANDFILL SITE BIOGAS Rania Ibrahim 1 , Shahram

BIOLOGICAL REMOVAL OF HYDROGEN SULPHIDE FROM LANDFILL SITE BIOGAS Rania Ibrahim 1 , Shahram

BIOLOGICAL REMOVAL OF HYDROGEN SULPHIDE FROM LANDFILL SITE BIOGAS Rania Ibrahim 1 , Shahram Navaee-Ardeh 1 , Hubert Cabana 1 1 Department of Civil Engineering, University of Sherbrooke, 2500 Boulevard de lUniversit, Sherbrooke, Qubec, J1K

265 views • 15 slides
What the E c onomists Say & What it Me ans to VB 2 Ge tting Be tte r Slowly!!!!

What the E c onomists Say & What it Me ans to VB 2 Ge tting Be tte r Slowly!!!!

What the E c onomists Say & What it Me ans to VB 2 Ge tting Be tte r Slowly!!!! Source: Bureau of Economic Analysis and Old Dominion Economists Growth Post- Re c e ssion Re a l GRP Bla c ksbur g - 2.36 Christia nsburg - Ra dfor

1.2k views • 45 slides
Training Compliance Claims Processing Health and Welfare Benefit Enrollment

Training Compliance Claims Processing Health and Welfare Benefit Enrollment

ADMINISTRATIVE PAYROLL INTERNAL SERVICES Purchasing & Payment Time & Attendance Risk Management Compliance (ADA, EEOC, FMLA) FLSA Compliance Property, Casualty & Employee Recognition IRS Payroll Reporting

821 views • 64 slides
ST. LUKE S ANGLICAN CHURCH, PALERMO Sunday, 2 February, 2020 10:30 a.m. Holy Eucharist

ST. LUKE S ANGLICAN CHURCH, PALERMO Sunday, 2 February, 2020 10:30 a.m. Holy Eucharist

ST. LUKE S ANGLICAN CHURCH, PALERMO Sunday, 2 February, 2020 10:30 a.m. Holy Eucharist Presentation of the Lord Presider: The Rev. Garfjeld Wu For emergency pastoral care, please call The Rev. Garfield Wu s cell phone number: 647-878-8650 . A

260 views • 4 slides
1 Image from clipartxtras.com 2 3 Sikhism is the fifth largest world religion. (If the students

1 Image from clipartxtras.com 2 3 Sikhism is the fifth largest world religion. (If the students

------Guide on how to use this presentation-------- In the notes section for each slide, we have provided talking points. In addition, we provided additional bullets for presentations that may allow for more time. We highly encourage you to

438 views • 20 slides
Investor Presentation December 2018 NYSE: DVN devonenergy.com Investor Contacts & Notices

Investor Presentation December 2018 NYSE: DVN devonenergy.com Investor Contacts & Notices

Investor Presentation December 2018 NYSE: DVN devonenergy.com Investor Contacts & Notices additional reserves; the uncertainties, costs and risks involved in oil and gas operations; regulatory restrictions, compliance costs and Investor

365 views • 24 slides

More recommend