deep learning
play

DEEP LEARNING FFR135, Artificial Neural Networks Olof Mogren - PowerPoint PPT Presentation

Nov 14, 2023 •200 likes •824 views

DEEP LEARNING FFR135, Artificial Neural Networks Olof Mogren Chalmers University of Technology October 2016 DEEP LEARNING Artificial neural networks Many layers of abstractions Outperforms traditional methods in: Image

  1. DEEP LEARNING FFR135, Artificial Neural Networks Olof Mogren Chalmers University of Technology October 2016

  2. DEEP LEARNING • Artificial neural networks • Many layers of abstractions • Outperforms traditional methods in: • Image classification • Natural language processing • Machine translation • Sentiment analysis • Speech recognition • Reinforcement learning

  3. SEMI-RECENT PROGRESS • 2006: Depth breakthrough: layerwise pretrained Restricted Boltzmann Machines • GPUs • Practical use Real applications from Google, Facebook, Tesla, Microsoft, Apple, and others! A fast learning algorithm for deep belief nets ; Hinton, Osindero, Tehi; Neural Computation; 2006

  4. PERCEPTRON • 1943, M cCulloch & Pitts (neuron model) inputs output • 1958, Rosenblatt (perceptron) x • Linear (binary) classification of inputs 0 w 0 x w • Can not learn any non-linear function 1 1 w (e.g. XOR) 2 y x 2 w 3 x w 3 4 x 4

  5. MODELLING XOR 1 1 0 x 0 0 0 1 0 1 x 1

  6. MODELLING XOR 1 1 0 1 1 x 0 x 0 ∧ ¬ x 1 0 0 1 0 0 1 0 1 0 1 ¬ x 0 ∧ x 1 x 1

  7. MULTI-LAYER PERCEPTRON • Combining layers lets us inputs hidden layer outputs represent non-linear functions • Each layer: • Linear transformation: a = W x + b • Non-linear (element-wise) activation: h = g ( a )

  8. MODELLING FUNCTIONS • Universal function approximation inputs hidden layer outputs • Stacking layers: function composition • Apply error/loss function to output • Continuously differentiable; chain rule • Propagating errors (backpropagation) • (Mini-batch) Stochastic gradient descent (SGD) details

  9. MOTIVATION OF DEPTH • M ore compact representation (exponentially) • There are boolean functions that require • Polynomial number of units ( deep architecture) • Exponential number of units ( shallow architecture) • E.g., parity function (for n input bits): • efficiently represented with depth O ( log n ) • but O ( 2 n ) gates if represented by a depth two circuit (Yao, 1985) Exploring Strategies for Training Deep Neural Networks ; Larochelle, Bengio, Louradour, Lamblin; JMLR 2009

  10. LEARNING LEVELS OF REPRESENTATION • E ach layer: non-linear transformation of inputs: h = sigmoid ( W x + b ) • Learning representations; abstractions • No feature engineering!

  11. DISTRIBUTED REPRESENTATIONS • E.g.: big, yellow, Volkswagen • Non-distributed representations: n binary parameters → n values • E.g.: Clustering, n-grams, decision trees, etc. • NNs learn distributed representations • Distributed representations: n binary parameters → 2 n possible values

  12. EXAMPLE: WORD EMBEDDINGS • Distributed representations for words • word2vec, glove, etc.

  13. DEEP LEARNING IN JAVASCRIPT cs231n.stanford.edu playground.tensorflow.org

  14. LEVELS OF ABSTRACTIONS

  15. Convolution Layer 32x32x3 image height 32 width 32 depth 3 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 10 27 Jan 2016 27 Jan 2016

  16. Convolution Layer 32x32x3 image 5x5x3 filter 32 Convolve the filter with the image i.e. “slide over the image spatially, computing dot products” 32 3 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 11 27 Jan 2016 27 Jan 2016

  17. Convolution Layer Filters always extend the full depth of the input volume 32x32x3 image 5x5x3 filter 32 Convolve the filter with the image i.e. “slide over the image spatially, computing dot products” 32 3 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 12 27 Jan 2016 27 Jan 2016

  18. Convolution Layer 32x32x3 image 5x5x3 filter 32 1 number: the result of taking a dot product between the filter and a small 5x5x3 chunk of the image 32 (i.e. 5*5*3 = 75-dimensional dot product + bias) 3 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 13 27 Jan 2016 27 Jan 2016

  19. Convolution Layer activation map 32x32x3 image 5x5x3 filter 32 28 convolve (slide) over all spatial locations 28 32 3 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 14 27 Jan 2016 27 Jan 2016

  20. consider a second, green filter Convolution Layer activation maps 32x32x3 image 5x5x3 filter 32 28 convolve (slide) over all spatial locations 28 32 3 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 15 27 Jan 2016 27 Jan 2016

  21. For example, if we had 6 5x5 filters, we’ll get 6 separate activation maps: activation maps 32 28 Convolution Layer 28 32 3 6 We stack these up to get a “new image” of size 28x28x6! Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 16 27 Jan 2016 27 Jan 2016

  22. Preview: ConvNet is a sequence of Convolution Layers, interspersed with activation functions 32 28 CONV, ReLU e.g. 6 5x5x3 32 28 filters 3 6 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 17 27 Jan 2016 27 Jan 2016

  23. Preview: ConvNet is a sequence of Convolutional Layers, interspersed with activation functions 32 28 24 …. CONV, CONV, CONV, ReLU ReLU ReLU e.g. 6 e.g. 10 5x5x3 5x5x 6 32 28 24 filters filters 3 6 10 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 18 27 Jan 2016 27 Jan 2016

  24. one filter => example 5x5 filters one activation map (32 total) We call the layer convolutional because it is related to convolution of two signals: elementwise multiplication and sum of a filter and the signal (image) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 21 27 Jan 2016 27 Jan 2016

  25. preview: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 22 27 Jan 2016 27 Jan 2016

  26. A closer look at spatial dimensions: activation map 32x32x3 image 5x5x3 filter 32 28 convolve (slide) over all spatial locations 28 32 3 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 23 27 Jan 2016 27 Jan 2016

  27. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 24 27 Jan 2016 27 Jan 2016

  28. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 25 27 Jan 2016 27 Jan 2016

  29. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 26 27 Jan 2016 27 Jan 2016

  30. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 27 Jan 2016 27 Jan 2016

  31. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter => 5x5 output 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 28 27 Jan 2016 27 Jan 2016

  32. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 29 27 Jan 2016 27 Jan 2016

  33. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 30 27 Jan 2016 27 Jan 2016

  34. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 => 3x3 output! 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 31 27 Jan 2016 27 Jan 2016

  35. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 3? 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 32 27 Jan 2016 27 Jan 2016

  36. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 3? doesn’t fit! 7 cannot apply 3x3 filter on 7x7 input with stride 3. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 33 27 Jan 2016 27 Jan 2016

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides
ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY MEDIA & ENTERTAINMENT SECURITY & DEFENSE AUTONOMOUS MACHINES Image

307 views • 26 slides
Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice Scaling up DL 2 What is Deep Learning? 3 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY SECURITY & DEFENSE MEDIA &

1.45k views • 39 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI Mission Helping people solve challenging problems using AI and deep learning. Developers, data scientists and engineers Self-driving cars,

1.44k views • 72 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin Clark Deep Learning for NLP 5 years ago No Seq2Seq No Attention No large-scale QA/reading comprehension datasets No TensorFlow or

1.41k views • 92 slides
An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning Microsoft Research Asia (MSRA) Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arXiv

1.32k views • 54 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning with multi-layer (3~30) neural networks, on a huge training set. State-of-the-art on many AI tasks Computer Vision : Image Classification,

1.53k views • 23 slides
Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Tofu: Parallelizing Deep Learning Systems with Automatic Tiling Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL system is based on dataflow GPU#0 w1 w2 data g1 g2 Forward propagation

405 views • 38 slides
Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides
Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu 2020 For the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline Deep Learning Growth, Celebrations, and Limitations

1.1k views • 87 slides
12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Wanli Ouyang, Zsolt Kira, Lawrence Neal, Raymond Yeh, Junting Lou and Teck-Yian Lim Unsupervised Learning in General Unsupervised learning is

1.48k views • 115 slides
Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-27 Definition Regularization is any modification we make to a learning algorithm that is intended to reduce

179 views • 13 slides
Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples RNNs Some slides borrowed from Fei-Fei Li & Justin Johnson &

1.1k views • 55 slides
Understanding Convolutional Neural Networks David Stutz July 24th, 2014 David Stutz | July

Understanding Convolutional Neural Networks David Stutz July 24th, 2014 David Stutz | July

Understanding Convolutional Neural Networks Understanding Convolutional Neural Networks David Stutz July 24th, 2014 David Stutz | July 24th, 2014 David Stutz | July 24th, 2014 0/53 1/53 Table of Contents - Table of Contents 1 Motivation

1.31k views • 72 slides
Deep Learning: Training Juhan Nam Training Deep Neural Networks Forward (hidden unit

Deep Learning: Training Juhan Nam Training Deep Neural Networks Forward (hidden unit

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Deep Learning: Training Juhan Nam Training Deep Neural Networks Forward (hidden unit activation) Parametric Gradient-based Learning Backward Non-parametric (gradient

650 views • 21 slides
Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More robust modular Dont need precise model Class-conditional densities are specification, so long as it is likely to be local, characteristic

1.09k views • 15 slides
CSI5180. MachineLearningfor BioinformaticsApplications Deep learning practical issues by

CSI5180. MachineLearningfor BioinformaticsApplications Deep learning practical issues by

CSI5180. MachineLearningfor BioinformaticsApplications Deep learning practical issues by Marcel Turcotte Version November 19, 2019 Preamble 2/31 salpha Preamble Preamble 2/31 Preamble Deep learning practical issues In this last

642 views • 60 slides
tss

tss

tr t tr rr rr

637 views • 24 slides
Le Lecture 7 7 R Recap ap I2DL: Prof. Niessner, Prof. Leal-Taix 1 Na Nave L Losse

Le Lecture 7 7 R Recap ap I2DL: Prof. Niessner, Prof. Leal-Taix 1 Na Nave L Losse

Le Lecture 7 7 R Recap ap I2DL: Prof. Niessner, Prof. Leal-Taix 1 Na Nave L Losse sses: s: L L2 v vs L s L1 L2 Loss: L1 Loss: $ = "#$ % ! = "#$ ! | " ( " )| %

973 views • 62 slides
Lectu ture 7 Recap Prof. Leal-Taix and Prof. Niessner 1 Bey Beyon ond l linea ear

Lectu ture 7 Recap Prof. Leal-Taix and Prof. Niessner 1 Bey Beyon ond l linea ear

Lectu ture 7 Recap Prof. Leal-Taix and Prof. Niessner 1 Bey Beyon ond l linea ear 1-layer network: f = Wx x W f 10 128128 Prof. Leal-Taix and Prof. Niessner 2 Ne Neural Ne Netw twork Width Depth Prof. Leal-Taix and

1.28k views • 92 slides
Neural Networks. Petr Pok Czech Technical University in Prague Faculty of Electrical

Neural Networks. Petr Pok Czech Technical University in Prague Faculty of Electrical

CZECH TECHNICAL UNIVERSITY IN PRAGUE Faculty of Electrical Engineering Department of Cybernetics Neural Networks. Petr Pok Czech Technical University in Prague Faculty of Electrical Engineering Dept. of Cybernetics P. Pok c 2017

1.09k views • 54 slides

More recommend