deep learning
play

Deep Learning T HEORY , H ISTORY , S TATE OF THE A RT & P - PowerPoint PPT Presentation

Dec 03, 2022 •146 likes •1.37k views

Deep Learning T HEORY , H ISTORY , S TATE OF THE A RT & P RACTICAL T OOLS by Ilya Kuzovkin ilya.kuzovkin@gmail.com Machine Learning Estonia http://neuro.cs.ut.ee 2016 Where it has started How it learns How it evolved What is the state

  1. NET OUT How it learns Backpropagation Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example

  2. NET OUT How it learns Backpropagation Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights Learning rate http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example

  3. NET OUT How it learns Backpropagation Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights Learning rate Gradient descent update rule http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example

  4. NET OUT How it learns Backpropagation Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example

  5. NET OUT How it learns Backpropagation Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 • Repeat for w6, w7, w8 http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example

  6. NET OUT How it learns Backpropagation Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 • Repeat for w6, w7, w8 • In analogous way for w1, w2, w3, w4 http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example

  7. NET OUT How it learns Backpropagation Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 • Repeat for w6, w7, w8 • In analogous way for w1, w2, w3, w4 • Calculate the total error again: 0.29 1027924 it was: 0.29 8371109 http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example

  8. NET OUT How it learns Backpropagation Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 • Repeat for w6, w7, w8 • In analogous way for w1, w2, w3, w4 • Calculate the total error again: 0.29 1027924 it was: 0.29 8371109 � • Repeat 10,000 times: 0.000035085 http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example

  9. How it learns Optimization methods Alec Radford “Introduction to Deep Learning with Python”

  10. How it learns Optimization methods Alec Radford “Introduction to Deep Learning with Python”

  11. How it evolved

  12. How it evolved 1-layer NN INPUT OUTPUT Alec Radford “Introduction to Deep Learning with Python”

  13. How it evolved 1-layer NN 92.5% on the MNIST test set INPUT OUTPUT Alec Radford “Introduction to Deep Learning with Python”

  14. How it evolved 1-layer NN 92.5% on the MNIST test set INPUT OUTPUT Alec Radford “Introduction to Deep Learning with Python”

  15. How it evolved One hidden layer Alec Radford “Introduction to Deep Learning with Python”

  16. How it evolved One hidden layer 98.2% on the MNIST test set Alec Radford “Introduction to Deep Learning with Python”

  17. How it evolved One hidden layer Activity of a 100 hidden 98.2% on the MNIST test set neurons (out of 625) Alec Radford “Introduction to Deep Learning with Python”

  18. How it evolved Overfitting

  19. How it evolved Dropout Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, 2014

  20. How it evolved Dropout Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, 2014

  21. How it evolved Dropout Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, 2014

  22. How it evolved Dropout Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, 2014

  23. How it evolved ReLU X. Glorot, A. Bordes, Y. Bengio, “Deep Sparse Rectifier Neural Networks”, 2011

  24. How it evolved ReLU X. Glorot, A. Bordes, Y. Bengio, “Deep Sparse Rectifier Neural Networks”, 2011

  25. How it evolved “Modern” ANN • Several hidden layers • ReLU activation units • Dropout

  26. How it evolved “Modern” ANN • Several hidden layers • ReLU activation units • Dropout

  27. How it evolved “Modern” ANN • Several hidden layers • ReLU activation units • Dropout 99.0% on the MNIST test set

  28. How it evolved Convolution

  29. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector

  30. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector

  31. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector

  32. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt 40 40 40 edge 40 40 40 detector 40 40 40 10 10 10 10 10 10 10 10 10

  33. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt +1 +1 +1 40 40 40 edge 0 0 0 40 40 40 detector -1 -1 -1 40 40 40 10 10 10 10 10 10 10 10 10

  34. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt +40 +40 +40 edge 0 0 0 detector -40 -40 -40 10 10 10 10 10 10 10 10 10

  35. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt +40 +40 +40 edge 0 0 0 0 detector -40 -40 -40 10 10 10 10 10 10 10 10 10

  36. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt 40 40 40 edge 0 40 40 40 detector 90 40 40 40 90 10 10 10 0 10 10 10 10 10 10

  37. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt 40 40 40 edge 0 40 40 40 detector 90 40 40 40 90 10 10 10 0 10 10 10 10 10 10

  38. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt 40 40 40 edge 0 40 40 40 detector 90 40 40 40 90 10 10 10 0 10 10 10 10 10 10

  39. How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt 40 40 40 edge 0 40 40 40 detector 90 40 40 40 90 10 10 10 0 10 10 10 10 10 10 Edge detector is a handcrafted feature detector.

  40. How it evolved Convolution The idea of a convolutional layer is to learn feature detectors instead of using handcrafted ones

  41. How it evolved Convolution The idea of a convolutional layer is to learn feature detectors instead of using handcrafted ones http://yann.lecun.com/exdb/mnist/

  42. How it evolved Convolution The idea of a convolutional layer is to learn feature detectors instead of using handcrafted ones 99.50% on the MNIST test set CURRENT BEST : 99.77% by committee of 35 conv. nets http://yann.lecun.com/exdb/mnist/

  43. How it evolved More layers

  44. How it evolved More layers C. Szegedy, et al., “Going Deeper with Convolutions”, 2014

  45. How it evolved More layers C. Szegedy, et al., “Going Deeper with Convolutions”, 2014 ILSVRC 2015 winner — 152 (!) layers K. He et al., “Deep Residual Learning for Image Recognition”, 2015

  46. How it evolved Hyperparameters • Network: architecture • number of layers • number of units (in each layer) • type of the activation function • weight initialization • • Convolutional layers: size • stride • number of filters • • Optimization method: learning rate • other method-specific • constants … •

  47. How it evolved Grid search :( Hyperparameters • Network: architecture • number of layers • number of units (in each layer) • type of the activation function • weight initialization • • Convolutional layers: size • stride • number of filters • • Optimization method: learning rate • other method-specific • constants … •

  48. How it evolved Grid search :( Hyperparameters Random search :/ • Network: architecture • number of layers • number of units (in each layer) • type of the activation function • weight initialization • • Convolutional layers: size • stride • number of filters • • Optimization method: learning rate • other method-specific • constants … •

  49. How it evolved Grid search :( Hyperparameters Random search :/ • Network: architecture • Bayesian optimization :) number of layers • number of units (in each layer) • type of the activation function • weight initialization • • Convolutional layers: size • stride • number of filters • • Optimization method: learning rate • other method-specific • constants … • Snoek, Larochelle, Adams, “Practical Bayesian Optimization of Machine Learning Algorithms”

  50. How it evolved Grid search :( Hyperparameters Random search :/ • Network: architecture • Bayesian optimization :) number of layers • number of units (in each layer) • type of the activation function • weight initialization • • Convolutional layers: size • stride • number of filters • • Optimization method: learning rate • other method-specific • constants Informal parameter search :) … • Snoek, Larochelle, Adams, “Practical Bayesian Optimization of Machine Learning Algorithms”

  51. How it evolved Major Types of ANNs feedforward convolutional

  52. How it evolved Major Types of ANNs feedforward convolutional recurrent

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides
ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY MEDIA & ENTERTAINMENT SECURITY & DEFENSE AUTONOMOUS MACHINES Image

307 views • 26 slides
Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice Scaling up DL 2 What is Deep Learning? 3 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY SECURITY & DEFENSE MEDIA &

1.45k views • 39 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI Mission Helping people solve challenging problems using AI and deep learning. Developers, data scientists and engineers Self-driving cars,

1.44k views • 72 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin Clark Deep Learning for NLP 5 years ago No Seq2Seq No Attention No large-scale QA/reading comprehension datasets No TensorFlow or

1.41k views • 92 slides
An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning Microsoft Research Asia (MSRA) Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arXiv

1.32k views • 54 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning with multi-layer (3~30) neural networks, on a huge training set. State-of-the-art on many AI tasks Computer Vision : Image Classification,

1.53k views • 23 slides
Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Tofu: Parallelizing Deep Learning Systems with Automatic Tiling Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL system is based on dataflow GPU#0 w1 w2 data g1 g2 Forward propagation

405 views • 38 slides
Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides
Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu 2020 For the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline Deep Learning Growth, Celebrations, and Limitations

1.1k views • 87 slides
12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Wanli Ouyang, Zsolt Kira, Lawrence Neal, Raymond Yeh, Junting Lou and Teck-Yian Lim Unsupervised Learning in General Unsupervised learning is

1.48k views • 115 slides
Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-27 Definition Regularization is any modification we make to a learning algorithm that is intended to reduce

179 views • 13 slides
Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples RNNs Some slides borrowed from Fei-Fei Li & Justin Johnson &

1.1k views • 55 slides
Activation Functions Activation Functions In [1]: % matplotlib inline import d2l from mxnet

Activation Functions Activation Functions In [1]: % matplotlib inline import d2l from mxnet

Activation Functions Activation Functions In [1]: % matplotlib inline import d2l from mxnet import autograd, nd def xyplot(x_vals, y_vals, name): d2l.set_figsize(figsize=(5, 2.5)) d2l.plt.plot(x_vals.asnumpy(), y_vals.asnumpy())

226 views • 7 slides
JACKPOT: Online Experimentation of Cloud Microservices BY M. TOSLALI 1 , S. PARTHASARATHY 2 , F.

JACKPOT: Online Experimentation of Cloud Microservices BY M. TOSLALI 1 , S. PARTHASARATHY 2 , F.

Jackpot: Online Experimentation of Cloud Microservices 7/15/2020 1 JACKPOT: Online Experimentation of Cloud Microservices BY M. TOSLALI 1 , S. PARTHASARATHY 2 , F. OLIVEIRA 2 , AND A. K. COSKUN 1 1 BOSTON UNIVERSITY; 2 IBM T.J. WATSON Talk

494 views • 18 slides
Dense layers IN TRODUCTION TO TEN S ORF LOW IN P YTH ON Isaiah Hull Economist The linear

Dense layers IN TRODUCTION TO TEN S ORF LOW IN P YTH ON Isaiah Hull Economist The linear

Dense layers IN TRODUCTION TO TEN S ORF LOW IN P YTH ON Isaiah Hull Economist The linear regression model INTRODUCTION TO TENSORFLOW IN PYTHON What is a neural network? INTRODUCTION TO TENSORFLOW IN PYTHON What is a neural network? A

715 views • 42 slides
CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of

CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of

CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Deep Neural Networks Gradient Vanishing Rectified linear units Overfitting

941 views • 25 slides
A general-purpose method for faithfully rounded floating-point function approximation in FPGAs

A general-purpose method for faithfully rounded floating-point function approximation in FPGAs

A general-purpose method for faithfully rounded floating-point function approximation in FPGAs David B. Thomas Imperial College London 1 David Thomas, Imperial College, dt10@ic.ac.uk FloPoCo : Parameterised primitives 2 David Thomas,

1.28k views • 81 slides
Gradient for Cross-Entropy Loss with Sigmoid For a single example ( x , y ): K

Gradient for Cross-Entropy Loss with Sigmoid For a single example ( x , y ): K

Gradient for Cross-Entropy Loss with Sigmoid For a single example ( x , y ): K L 1 L y k log k ( x ) + (1 y k ) log k ( x ) k =1 s l 1 L s l + 2 w l ij 2 m l =1 i =1

912 views • 50 slides
EN. 601.467/667 Introduction to Human Language Technology Deep Learning II Shinji Watanabe 1

EN. 601.467/667 Introduction to Human Language Technology Deep Learning II Shinji Watanabe 1

EN. 601.467/667 Introduction to Human Language Technology Deep Learning II Shinji Watanabe 1 Todays agenda Basics of (deep) neural network How to integrate DNN with HMM Recurrent neural network Language modeling

1.33k views • 60 slides
Logistic Regression: From Binary to Multi-Class Shuiwang Ji Department of Computer Science &

Logistic Regression: From Binary to Multi-Class Shuiwang Ji Department of Computer Science &

Logistic Regression: From Binary to Multi-Class Shuiwang Ji Department of Computer Science & Engineering Texas A&M University 1 / 14 Binary Logistic Regression 1 The binary LR predicts the label y i { 1 , +1 } for a given sample

306 views • 14 slides

More recommend