Parallelized Training of Deep NN Comparison of Current Concepts and - PowerPoint PPT Presentation

Nov 30, 2022 •357 likes •515 views

Parallelized Training of Deep NN Comparison of Current Concepts and Frameworks Sebastian Jger, Hans-Peter Zorn, Stefan Igel, Christian Zirpins Rennes, Dec 10, 2018 Motivation Need to scale the training of neural networks horizontally

Parallelized Training of Deep NN Comparison of Current Concepts and Frameworks Sebastian Jäger, Hans-Peter Zorn, Stefan Igel, Christian Zirpins Rennes, Dec 10, 2018
Motivation › Need to scale the training of neural networks horizontally › Kubernetes based technology stack › Scalability of concepts and frameworks 2
Distributed Training Methods Data Parallelism 3
Data Parallelism Centralized Parameter Server TensorFlow: https://www.tensorflow.org 4
Data Parallelism Decentralized Parameter Server Apache MXNet: http://mxnet.apache.org 5
Experimental Setup Environment › Google Kubernetes Engine › CPU: 2.6 GHz › Ubuntu 16.04 › TensorFlow 1.8.0 › MXNet 1.3.0 6
Experimental Setup Networks Convolutional NN Recurrent NN › LeNet-5 › LSTM › 5 layer › 2 layer › 10 classes › 200 units › Fashion MNIST › Penn Tree Bank › 28x28 gray-scale › 1.000.000 words 7
Experimental Setup Metrics 8
Results Convolutional Neural Network 9
Results Convolutional Neural Network 10
Results Recurrent Neural Network 11
Summarizating the Experiments Decentralized Parameter Server ... › more robust regarding increasing communication effort › scales better for small NN For bigger/ more complex NN … › no significant difference between concepts 12
Conclusion MXNet ... › for small NN better scalability and throughput › for bigger NN higher throughput › less and less complicated code › easier to scale up training 13
Thank you Sebastian Jäger @se_jaeger inovex GmbH Ludwig-Erhard-Allee 6 76131 Karlsruhe sebastian.jaeger@inovex.de

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Deep Learning: Training Juhan Nam Training Deep Neural Networks Forward (hidden unit

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Deep Learning: Training Juhan Nam Training Deep Neural Networks Forward (hidden unit activation) Parametric Gradient-based Learning Backward Non-parametric (gradient

650 views • 21 slides

BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding ( B idirectional

BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding ( B idirectional E ncoder R epresentations from T ransformers) Jacob Devlin Google AI Language Pre-training in NLP Word embeddings are the basis of deep

1.56k views • 37 slides

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides

Speeding R up on your computer by parallelized computations a geostatistical case study

Speeding R up on your computer by parallelized computations a geostatistical case study Andreas Papritz Department of Environmental Systems Science, ETH Zurich Statistik Stadt Zrich papritz@env.ethz.ch outline motivating example:

368 views • 34 slides

Normalization Techniques in Training of Deep Neural Networks Lei Huang ( ) State Key

Normalization Techniques in Training of Deep Neural Networks Lei Huang ( ) State Key Laboratory of Software Development Environment, Beihang University Mail:huanglei@nlsde.buaa.edu.cn August 17 th , 2017 Outline Introduction to Deep

845 views • 47 slides

Parallelized and Vectorized Tracking Using Kalman Filter with CMS Detector Geometry and Events G.

Parallelized and Vectorized Tracking Using Kalman Filter with CMS Detector Geometry and Events G. Cerati 4 , P. Elmer 3 , M. Kortelainen 4 , S. Krutelyov 1 , S. Lantz 2 , M. Lefebvre 3 , M. Masciovecchio 1 , K. McDermott 2 , B. Norris 5 , D. Riley

390 views • 23 slides

Optimization for Training Deep Models Xiaogang Wang xgwang@ee.cuhk.edu.hk February 12, 2019

Optimization Basics Optimization of training deep neural networks Multi-GPU Training Optimization for Training Deep Models Xiaogang Wang xgwang@ee.cuhk.edu.hk February 12, 2019 cuhk Xiaogang Wang Optimization for Training Deep Models

1.69k views • 72 slides

Parallelized RSA Algorithm: An Analysis With Performance Evaluation using OpenMP Library in High

Parallelized RSA Algorithm: An Analysis With Performance Evaluation using OpenMP Library in High Performance Computing Environment Md. Ahsan Ayub , Zishan Ahmed Onik, Steven Smith Department of Computer Science Tennessee Technological

507 views • 19 slides

affyPara: Parallelized preprocessing algorithms for high-density oligonucleotide array data

affyPara: Parallelized preprocessing algorithms for high-density oligonucleotide array data Markus Schmidberger Ulrich Mansmann UseR 2008 IBE August 12-14, Technische Universitt Dortmund, http://ibe.web.med.uni-muenchen.de Germany

359 views • 23 slides

Parallelized Kalman-Filter-Based Reconstruction of Particle Tracks with Accurate Detector

Parallelized Kalman-Filter-Based Reconstruction of Particle Tracks with Accurate Detector Geometry G. Cerati 4 , P. Elmer 3 , M. Kortelainen 4 , S. Krutelyov 1 , S. Lantz 2 , M. Lefebvre 3 , M. Masciovecchio 1 , K. McDermott 2 , D. Riley 2 ,

406 views • 26 slides

Parameter efficient training of deep convolutional neural networks by dynamic sparse

Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization Hesham Mostafa (Intel AI) Xin Wang (Intel AI, Cerebras Systems) Easy : post-training (sparse) compression Hard : direct training of sparse

727 views • 9 slides

Industrial Level Deep Learning Training Infrastructure the Practice and Experience from

Industrial Level Deep Learning Training Infrastructure the Practice and Experience from SenseTime Shengen Yan SenseTime Group Limited. The Success of Deep Learning Google Search AlexNet won ImageNet 2006-01 2007-01 2008-01 2009-01

498 views • 25 slides

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1 Quick Recap: Training a network Divergence between desired output and Average over all actual output of net for a given input training instances

1.75k views • 140 slides

Accelerate Deep Learning Training at Scale on GPUs Maggie Zhang ( ) maggiez@nvidia.com

Accelerate Deep Learning Training at Scale on GPUs Maggie Zhang ( ) maggiez@nvidia.com Introduction Why do we need to scale training AGENDA How to achieve scaling DL Training: from single GPU to multi-node 1.33 Minutes on

894 views • 47 slides

Paper number: 230 Training deep segmenta.on networks on texture- encoded input: applica.on to

Paper number: 230 Training deep segmenta.on networks on texture- encoded input: applica.on to neuroimaging of the developing neonatal brain AE AE Fe Fe&t, J Cupi., T Kart, D Rueckert The shape hypothesis in deep CNNs cat

476 views • 13 slides

A Parallelized Theorem Prover for Interactive Theorem Proving David L. Rager, Warren A. Hunt Jr.,

Introduction Parallelism Primitives Parallelizing ACL2 Evaluate Approach Conclusion A Parallelized Theorem Prover for Interactive Theorem Proving David L. Rager, Warren A. Hunt Jr., and Matt Kaufmann ragerdl@defthm.com, hunt@cs.utexas.edu,

835 views • 59 slides

Goutham Veeramachaneni @putadent Me Goutham Veeramachaneni | 2 Debugging slow queries Cortex |

696 views • 26 slides

Apache Solr la piattaforma di ricerca enterprise LucaBonesini | Titulus User Group, Kion

Apache Solr la piattaforma di ricerca enterprise LucaBonesini | Titulus User Group, Kion Bologna 4/dic/2013 Chi s hi son ono Luca Bonesini Infor orma matico Lanciatore di giavellotti Prog ogramma mmator ore Suonatore di chitarra

986 views • 39 slides

REST... with Peace Content Management with Apache Sling 1 Freitag, 25. Februar 2011 The Problem

REST... with Peace Content Management with Apache Sling 1 Freitag, 25. Februar 2011 The Problem (abstract) Store large amounts of different types of content Associate meta data to content Access control Search for stuff

539 views • 16 slides

JCR, Jackrabbit, Jackalope Liip Techday 1. July 2009 Tobias Ebnther & Christian Stocker

JCR, Jackrabbit, Jackalope Liip Techday 1. July 2009 Tobias Ebnther & Christian Stocker Jackalope What is a Content Repository A thing to store and retrieve content Hierarchical -> See CRX Explorer Jackalope Content

709 views • 14 slides

The Moon as a Detector for Extreme-Energy Cosmic Rays

The Moon as a Detector for Extreme-Energy Cosmic Rays Hallsie Reno University of Iowa March 7, 2013 Jeong, Reno and Sarcevic, AstroparIcle Physics

418 views • 25 slides

The politics of helping the poor Lane Kenworthy June 2010 Economic growth has been good for the

The politics of helping the poor Lane Kenworthy June 2010 Economic growth has been good for the poor Denmark Sweden 13,000 13,000 P10 income (2000 US$ P10

794 views • 37 slides

Service Mess to Service Mesh Observe. Control. Secure. Rob Richardson Technical Evangelist,

Service Mess to Service Mesh Observe. Control. Secure. Rob Richardson Technical Evangelist, MemSQL Kavya Pearlman Cybersecurity Strategist, Wallarm https://www.shutterstock.com/image-photo/ca r-technology-autonomous-self-driving-concep

306 views • 26 slides

Vector Flows and integer flows Yi Wang School of mathematical sciences, Anhui University Joint

Introduction to integer flow 5-flow Conjecture and 3-flow Conjecture Vector flow Vector Flows and integer flows Yi Wang School of mathematical sciences, Anhui University Joint with Jian Cheng, Rong Luo and Cun-Quan Zhang Dec. 20, 2016,

639 views • 25 slides