dee deep le p learni arning ng mod models els
play

DEE DEEP LE P LEARNI ARNING NG MOD MODELS ELS Mathew Salvaris - PowerPoint PPT Presentation

Jan 18, 2024 •298 likes •610 views

DIS DISTRI TRIBUT BUTED ED TRA TRAINI INING NG OF OF DEE DEEP LE P LEARNI ARNING NG MOD MODELS ELS Mathew Salvaris @msalvaris Ilia Karmanov @ikdeepl Miguel Fierro @miguelgfierro Rosetta Stone of

  1. DIS DISTRI TRIBUT BUTED ED TRA TRAINI INING NG OF OF DEE DEEP LE P LEARNI ARNING NG MOD MODELS ELS Mathew Salvaris @msalvaris Ilia Karmanov @ikdeepl Miguel Fierro @miguelgfierro

  2. Rosetta Stone of Deep Learning more info: https://github.com/ilkarman/DeepLearningFrameworks Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  3. ImageNet Competition error (%) ImageNet top-5 error 15.3% 7.3% 6.7% 5.1% (human) 3.8% 3.8% 3.6% 3.1% 2.4% Inception- ResNext ResNet NASNet AmoebaNet AlexNet VGG Inception ResNet Instagram (2015) (2017) (2017) (2012) (2014) (2015) (2016) (2018) Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  4. Distributed training mode: Data parallelism Worker 1 Worker 2 Job manager CNN model Subset 1 CNN model Subset 2 CNN model Dataset Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  5. Distributed training mode: Model parallelism Worker 1 Worker 2 Job manager Submodel 1 Submodel 2 CNN model Dataset Dataset Dataset Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  6. Data parallelism vs model parallelism Da Data ta parallelism arallelism Mod odel el parallelism arallelism ▪ Easier implementation ▪ Better scalability of large models ▪ Stronger fault tolerance ▪ Less memory on each GPU ▪ Higher cluster utilization Why no Wh y not bo t both th? ? Da Data ta para aralleli llelism sm fo for r CN CNN lay N layers ers an and model del par aralle allelism lism in in FC FC la laye yers rs so source: ce: Alex ex Krizhevs zhevsky ky. . 2014. On One weird rd trick ck fo for pa paralleli lelizing zing co convolutio lutional nal neura ural l netwo works. rks. https ps://a //arxiv.o rxiv.org/a rg/abs bs/14 /1404.5 04.5997 Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  7. Training strategies: parameter averaging Worker 1 Worker 2 Subset 1 CNN model Subset 2 CNN model Average of weights for each worker Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  8. Training strategies: distributed gradient based Worker 1 Worker 2 Subset 1 CNN model Subset 2 CNN model Synchronous Gradients of each worker Asynchronous Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  9. Overview of distributed training Install software and containers Schedule jobs Provision clusters of VMs Share results Distribute data Scale resources Handling failures Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  10. Azure Distributed Platforms ▪ Batch AI ▪ Batch Shipyard ▪ DL Workspace Horovod Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  11. Batch Shipyard • Supports Docker and Singularity: run your Docker and Singularity containers within the same job, side-by-side or even concurrently • Move data easily between locally accessible storage systems, remote filesystems, Azure Blob or File Storage, and compute nodes • Supports local storage, Azure Blob or File Storage, and NFS. • Low priority nodes https://github.com/Azure/batch-shipyard Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  12. Batch AI • Supports running on Docker container as well as the Data Science Virtual Machine • Supports local storage, Azure Blob or File Storage, and NFS. • Low priority nodes https://github.com/Azure/BatchAI Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  13. DL Workspace • Runs jobs inside Docker • Uses Kubernetes • Can be deployed anywhere not just Azure • Supports local storage and NFS https://github.com/Microsoft/DLWorkspace Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  14. Training with Batch AI 1) Create scripts to run on Batch AI 1 and transfer them to file storage I 2) Write the data to storage A I 2 3) Create the docker containers for each DL framework and transfer them to a container registry 3 Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  15. 1) Create a Batch AI Pool I A I 2 2) Each job will pull in the 2 appropriate container, script and 1 Batch AI Pool load data from chosen storage 3) Once the job is completed all the 2 results will be written to the fileshare 3 Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  16. Batch AI Interface CLI Python SDK az batchai cluster create --name nc24r --image UbuntuLTS --vm-size Standard_NC24rs_v3 --min 8 --max 8 --afs-name $FILESHARE_NAME --afs-mount-path extfs --storage-account-name $STORAGE_ACCOUNT_NAME --storage-account-key $storage_account_key --nfs $NFS_NAME --nfs-mount-path nfs Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  17. Distributed training with NFS ▪ Batch AI cluster configuration with Copy Data NFS share Batch AI Pool NFS I Share A I az batchai cluster create Mounted --name nc24r Fileshare --image UbuntuLTS --vm-size Standard_NC24rs_v3 --min 8 --max 8 --afs-name $FILESHARE_NAME --afs-mount-path extfs --storage-account-name $STORAGE_ACCOUNT_NAME --storage-account-key $storage_account_key --nfs $NFS_NAME --nfs-mount-path nfs Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  18. Distributed training with blob storage ▪ Batch AI cluster configuration with Copy Data mounted blob Batch AI Pool Mounted I Blob A I az batchai cluster create Mounted --name nc24r Fileshare --image UbuntuLTS --vm-size Standard_NC24rs_v3 --min 8 --max 8 --afs-name $FILESHARE_NAME --afs-mount-path extfs --container-name $CONTAINER_NAME --container-mount-path extcn --storage-account-name $STORAGE_ACCOUNT_NAME --storage-account-key $storage_account_key Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  19. Distributed training with local storage ▪ Batch AI cluster configuration with Copy Data copying the data to the nodes Batch AI Pool I A I az batchai cluster create --name nc24r Mounted --image UbuntuLTS Fileshare --vm-size Standard_NC24r --min 8 --max 8 --afs-name $FILESHARE_NAME --afs-mount-path extfs --container-name $CONTAINER_NAME --container-mount-path extcn --storage-account-name $STORAGE_ACCOUNT_NAME --storage-account-key $storage_account_key Node preparation configuration -c cluster.json Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  20. Distributed training Results images/second Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  21. Distributed training Results images/second Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  22. Distributed training Results images/second Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  23. Distributed training with Horovod Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  24. Distributed training with Horovod Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  25. Distributed training with Horovod Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  26. Distributed training with PyTorch Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  27. Distributed training with Chainer Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  28. Distributed training with CNTK 1-bit SGD with MPI Blocked Momentum with MPI Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  29. Demo Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  30. Acknowledgements Hongzhi Li Alex Sutton Alex Yukhanov Attribution of some images: http://morguefile.com/ Mathew Salvaris (@msalvaris) – Ilia Karmanov (@ikdeepl) – Miguel Fierro (@miguelgfierro)

  31. Thanks! Mathew Salvaris @msalvaris Ilia Karmanov @ikdeepl Miguel Fierro @miguelgfierro

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

BUI UILDI DING NG A ROB OBUST ON ONLINE LEARNI ARNING NG S STRUCTURE ON RE ON THE OL

BUI UILDI DING NG A ROB OBUST ON ONLINE LEARNI ARNING NG S STRUCTURE ON RE ON THE OL

BUI UILDI DING NG A ROB OBUST ON ONLINE LEARNI ARNING NG S STRUCTURE ON RE ON THE OL OLC P PILLAR ARS OF OF QUAL ALITY BY HAP AZIZ, ED.D. TWITTER: @DIGITALHAP CONTACT: HAPAZIZ@GMAIL.COM BLOG: HTTP://HAPAZIZ.WORDPRESS.COM Session

1.04k views • 17 slides
Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go

Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go

Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go Going ful g full B Baye yesi sian Bayes = Probabilities Hypothesis = Model Bayes Theorem p ( H | E ) = p ( E | H ) p ( H ) p ( E )

776 views • 61 slides
Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go

Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go

Ba Bayesi esian Deep Deep Le Lear arning ning Prof. Leal-Taix and Prof. Niessner 1 Go Going ful g full Bay ayes esian an Bayes = Probabilities Hypothesis = Model Bayes Theorem Evidence = data Prof. Leal-Taix and Prof.

688 views • 66 slides
Excitation Deep Learning Model Abdul Qayyum ImViA laboratory, University of Bourgogne

Excitation Deep Learning Model Abdul Qayyum ImViA laboratory, University of Bourgogne

Medi dical cal Imagin ging g with th Deep ep Learni ning ng 2020 20 Short t Paper er MID IDL L 2020 20 Segmentation of the Myocardium on Late-Gadolinium Enhanced MRI based on 2.5 D Residual Squeeze and Excitation Deep Learning

203 views • 5 slides
Wi Winte ter r Prof ofessi essional onal Learni arning ng Conf nfer erence ence Febr

Wi Winte ter r Prof ofessi essional onal Learni arning ng Conf nfer erence ence Febr

Richard Woods, Georgias School Superintendent Educating Georgias Future gadoe.org Career eer, , Tec echni hnical cal and nd Agricu ricult ltur ure e Educ ucati tion on Wi Winte ter r Prof ofessi essional onal

595 views • 40 slides
Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications Neurons & Nets dimer Learning & Depth G1E model Feature extraction Theory 1 Layer Nets Neural computation

1.32k views • 45 slides
Motivation Issues of the current live based educational system (DVTS or Vidyo etc) to the

Motivation Issues of the current live based educational system (DVTS or Vidyo etc) to the

Nopphol Pausawasdi, Mahidol University Siriraj Hospital On On demand demand 4K 4K Telemedici dicine ne E learni arning ng Syste System in in Cloud Cloud En Envir vironm nment Nopphol Pausawasdi (Siriraj Hospital, Mahidol Univ. ,

409 views • 8 slides
ICES S TAND ARD - BAS ED S ERVICES NDARD ASED Mariane COMTE University of Montpellier r Event

ICES S TAND ARD - BAS ED S ERVICES NDARD ASED Mariane COMTE University of Montpellier r Event

.eu http:/ p://ww www.l .learni arning ng-com compa pass.e C OM ING L IFEL ONG L EARNIN RNING OMPOS OSIN ELON O PPOR NITY P ATHW HWAYS THRO GH ORTU TUNIT ROUGH ICES S TAND ARD - BAS ED S ERVICES NDARD ASED Mariane COMTE University

1.38k views • 69 slides
Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting Wang* 2 , Quanshi Zhang 1 , Runjin Chen 1 , Di He 2 , Xing Xie 2 * Equal Contribution 1 John Hopcroft Center and the MoE Key Lab of Artificial

219 views • 11 slides
RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab

RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab

RECURSIVE DEEP MODELS FOR SEMANTIC 1 COMPOSITIONALITY Zhicong Lu DGP Lab luzhc@dgp.toronto.edu 1 Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng and Christopher Potts. Recursive Deep Models for Semantic

518 views • 29 slides
BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For

BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For

BIL 722: Advanced Topics in Computer Vision Mehmet Kerim Y cel Deep Structured Models For Group Activity Recognition Deng et al. BMVC 2015 Simon Fraser University, SPORTLOGIQ , Canada Overview 25 April 2016 Deep Structured Models For

701 views • 40 slides
Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant recognition in the ventral stream Center Name Presenter Name Towards a theory of the above tomaso poggio, CBMM, BCS, CSAIL, McGovern MIT Plan

1.14k views • 81 slides
Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-10-04 Roadmap Challenges of Unstructured Modeling Using Graphs to Describe Model Structure

974 views • 34 slides
Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

A New Framework For Hybrid Models An Application: Deep Hybrid Models Supervised and Semi-Supervised Experiments Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and Stefano Ermon Department of Computer

523 views • 29 slides
Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models have shown promising results in many domains. However, such models often have millions of parameters. The deep learning models face the following

314 views • 16 slides
Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye

Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye

Hybrid Models with Deep and Invertible Features Eric Nalisnick *, Akihiro Matsukawa*, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan *equal contribution Predictive Models Hybrid Models with Deep and Invertible Features Predictive Models

809 views • 16 slides
Mason Public Schools Distance Learning Plan Recap Since March 13, 2020 In-person

Mason Public Schools Distance Learning Plan Recap Since March 13, 2020 In-person

Mason Public Schools Distance Learning Plan Recap Since March 13, 2020 In-person instruction was suspended per Executive Order 2020-35 beginning March 16, 2020. Staff provided at-home work for students from late March to April 17.

542 views • 15 slides
Custom Writing Service - Special Prices Phd dissertation proposal presentation Research paper

Custom Writing Service - Special Prices Phd dissertation proposal presentation Research paper

Custom Writing Service - Special Prices Phd dissertation proposal presentation Research paper about study habits books creative writing names story starters .. Critical thinking jokes assessment Critique of a research papers zoology elements of

484 views • 3 slides
Custom Writing Service - Special Prices Buy a dissertation defense presentation Research papers

Custom Writing Service - Special Prices Buy a dissertation defense presentation Research papers

Custom Writing Service - Special Prices Buy a dissertation defense presentation Research papers on love research paper on corporate governance writing a rationale for a dissertation derek swetnam marketing dissertation naoum conclusion on

418 views • 3 slides
TECHNIQUES FOR WRITING AND PRESENTATION OF THESIS: A COMPANION GUIDE TO POSTGRADUATE STUDENTS IN

TECHNIQUES FOR WRITING AND PRESENTATION OF THESIS: A COMPANION GUIDE TO POSTGRADUATE STUDENTS IN

TECHNIQUES FOR WRITING AND PRESENTATION OF THESIS: A COMPANION GUIDE TO POSTGRADUATE STUDENTS IN NIGERIAN UNIVERSITY SYSTEM BY UMAR IBRAHIM, BLS, MLS, PhD (ABU), FNLA, CLN Umarahim2003@yahoo.com. 08037022011 Being a paper presented at a

312 views • 12 slides
DISTRIBUTED STREAMING TEXT EMBEDDING METHOD => DISTRIBUTED TRAINING WITH PYTORCH SNU 2018 - 2

DISTRIBUTED STREAMING TEXT EMBEDDING METHOD => DISTRIBUTED TRAINING WITH PYTORCH SNU 2018 - 2

DISTRIBUTED STREAMING TEXT EMBEDDING METHOD => DISTRIBUTED TRAINING WITH PYTORCH SNU 2018 - 2 BIg Data and Deep Learning 2018. 12. 18 Final Project Team 1 , , , DISTRIBUTED STREAMING TEXT EMBEDDING

195 views • 17 slides
Distance Education Affinity Group: Best Practices for Online Learning Embracing our Students

Distance Education Affinity Group: Best Practices for Online Learning Embracing our Students

Distance Education Affinity Group: Best Practices for Online Learning Embracing our Students Educational Needs Through Technology NJEdge November 17, 2016 Panelists and Affiliations Dr. Leigh M. Bello-de Castro, Associate Dean- West

491 views • 16 slides
IS WRONG. Dave Birss Hello, Im Dave Birss prepared for Owen James - A Meeting of Minds

IS WRONG. Dave Birss Hello, Im Dave Birss prepared for Owen James - A Meeting of Minds

IF CREATIVITY IS THE ANSWER, THE QUESTION IS WRONG. Dave Birss Hello, Im Dave Birss prepared for Owen James - A Meeting of Minds Winning Advisers - by Dave Birss Creative stuff is my bag prepared for Owen James - A Meeting of

965 views • 53 slides
Presented By: Marianne Litzman , Assistant Superintendent for Curriculum & Instruction Susan

Presented By: Marianne Litzman , Assistant Superintendent for Curriculum & Instruction Susan

Presented By: Marianne Litzman , Assistant Superintendent for Curriculum & Instruction Susan Guiliano , Director of Curriculum, Instruction & Assessments Ronald Labrocca , Supervisor for Mathematics, Business & Computer Education Michael

961 views • 49 slides

More recommend