neural networks
play

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain - PowerPoint PPT Presentation

Jan 31, 2023 •528 likes •1.29k views

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What well cover ... f ( x ) computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ...

  1. Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain

  2. 2 NEURAL NETWORKS • What we’ll cover ... • f ( x ) ‣ computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ... ‣ natural language processing architectures - word embeddings - recurrent neural networks 1 ... ... - long short-term memory networks (LSTMs) 1 ... ... • x 1 x j x d x

  3. Neural Networks Computer vision

  4. 4 COMPUTER VISION Topics: computer vision, object recognition • Computer vision is the design of computers that can process visual data and accomplish some given task ‣ we will focus on object recognition: given some input image, identify which object it contains Caltech 101 dataset 112 pixels ‘‘sun flower’’ 150 pixels

  5. 5 COMPUTER VISION Topics: computer vision • We can design neural networks that are specifically adapted for such problems ‣ must deal with very high-dimensional inputs - 150 x 150 pixels = 22500 inputs, or 3 x 22500 if RGB pixels ‣ can exploit the 2D topology of pixels (or 3D for video data) ‣ can build in invariance to certain variations we can expect - translations, illumination, etc. • Convolutional networks leverage these ideas ‣ local connectivity ‣ parameter sharing ‣ pooling / subsampling hidden units

  6. 6 COMPUTER VISION Topics: local connectivity ... • First idea: use a local ... ... connectivity of hidden units ‣ each hidden unit is connected only to a subregion (patch) of the input image ‣ it is connected to all channels - 1 if greyscale image - 3 (R, G, B) for color image • Solves the following problems: ‣ fully connected hidden layer would have an unmanageable number of parameters r = receptive field ‣ computing the linear activations of the hidden units would be very expensive

  7. 7 COMPUTER VISION Topics: local connectivity • Units are connected to all channels: ‣ 1 channel if grayscale image, 3 channels (R, G, B) if color image ... ... ...

  8. 8 COMPUTER VISION Topics: parameter sharing • Second idea: share matrix of parameters across certain units ‣ units organized into the same ‘‘feature map’’ share parameters ‣ hidden units within a feature map cover different positions in the image feature map 1 feature map 2 feature map 3 ... ... ... ... ... ... same color = same matrix of connections 6

  9. 8 COMPUTER VISION Topics: parameter sharing • Second idea: share matrix of parameters across certain units ‣ units organized into the same ‘‘feature map’’ share parameters ‣ hidden units within a feature map cover different positions in the image feature map 1 feature map 2 feature map 3 ... ... ... ... ... ... same color = same matrix of connections 6

  10. 8 COMPUTER VISION Topics: parameter sharing • Second idea: share matrix of parameters across certain units ‣ units organized into the same ‘‘feature map’’ share parameters ‣ hidden units within a feature map cover different positions in the image feature map 1 feature map 2 feature map 3 ... ... ... ... ... ... same color = same matrix of connections 6

  11. 8 COMPUTER VISION Topics: parameter sharing • Second idea: share matrix of parameters across certain units ‣ units organized into the same ‘‘feature map’’ share parameters ‣ hidden units within a feature map cover different positions in the image feature map 1 feature map 2 feature map 3 ... ... ... ... ... ... same color = same matrix of connections 6

  12. 8 COMPUTER VISION Topics: parameter sharing • Second idea: share matrix of parameters across certain units ‣ units organized into the same ‘‘feature map’’ share parameters ‣ hidden units within a feature map cover different positions in the image feature map 1 feature map 2 feature map 3 ... ... ... ... ... ... W ij is the matrix same color connecting the i th = input channel with same matrix of connections the j th feature map 6

  13. 8 COMPUTER VISION Topics: parameter sharing • Second idea: share matrix of parameters across certain units ‣ units organized into the same ‘‘feature map’’ share parameters ‣ hidden units within a feature map cover different positions in the image feature map 1 feature map 2 feature map 3 ... ... ... ... ... ... W ij is the matrix same color connecting the i th = input channel with same matrix of connections the j th feature map 6

  14. 9 COMPUTER VISION Topics: parameter sharing • Solves the following problems: ‣ reduces even more the number of parameters ‣ will extract the same features at every position (features are ‘‘equivariant’’) feature map 1 feature map 2 feature map 3 ... ... ... ... ... ... W ij is the matrix same color connecting the i th = input channel with same matrix of connections the j th feature map

  15. 10 COMPUTER VISION Topics: parameter sharing Jarret et al. 2009 • Each feature map forms a 2D grid of features ‣ can be computed with a discrete convolution ( ) of a kernel matrix k ij which is 
 H X ∗ the hidden weights matrix W ij with its rows and columns flipped 
 f e a t u r e m ‣ x i is the i th channel of input a p s ‣ k ij is the convolution kernel ‣ g j is a learned scaling factor ‣ y j is the hidden layer (could have added a bias) � y j = g j tanh( k ij ∗ x i ) i

  16. 
 11 COMPUTER VISION Topics: discrete convolution • The convolution of an image x with a kernel k is computed as follows: 
 ( x * k ) ij = ∑ x i + p,j + q k r-p,r-q • Example: 0 80 40 0 0.25 = * 20 40 0 0.5 1 0 0 40 k x

  17. 
 11 COMPUTER VISION Topics: discrete convolution • The convolution of an image x with a kernel k is computed as follows: 
 ( x * k ) ij = ∑ x i + p,j + q k r-p,r-q pq • Example: ~ = k with rows and columns flipped k 0 80 40 1 0.5 0 0.25 = * 20 40 0 0.25 0 0.5 1 0 0 40 k x

  18. 
 12 COMPUTER VISION Topics: discrete convolution • The convolution of an image x with a kernel k is computed as follows: 
 ( x * k ) ij = ∑ x i + p,j + q k r-p,r-q pq • Example: 1 x 0 + 0.5 x 80 + 0.25 x 20 + 0 x 40 0 80 40 1 0.5 0 0.25 45 = * 20 40 0 0.25 0 0.5 1 0 0 40 k x

  19. 
 13 COMPUTER VISION Topics: discrete convolution • The convolution of an image x with a kernel k is computed as follows: 
 ( x * k ) ij = ∑ x i + p,j + q k r-p,r-q pq • Example: 1 x 80 + 0.5 x 40 + 0.25 x 40 + 0 x 0 0 80 40 1 0.5 0 0.25 45 110 = * 20 40 0 0.25 0 0.5 1 0 0 40 k x

  20. 
 14 COMPUTER VISION Topics: discrete convolution • The convolution of an image x with a kernel k is computed as follows: 
 ( x * k ) ij = ∑ x i + p,j + q k r-p,r-q pq • Example: 1 x 20 + 0.5 x 40 + 0.25 x 0 + 0 x 0 0 80 40 0 0.25 45 110 = * 20 40 0 1 0.5 0.5 1 40 0 0 40 k 0.25 0 x

  21. 
 15 COMPUTER VISION Topics: discrete convolution • The convolution of an image x with a kernel k is computed as follows: 
 ( x * k ) ij = ∑ x i + p,j + q k r-p,r-q pq • Example: 1 x 40 + 0.5 x 0 + 0.25 x 0 + 0 x 40 0 80 40 0 0.25 45 110 = * 20 40 0 1 0.5 0.5 1 40 40 0 0 40 k 0.25 0 x

  22. 16 COMPUTER VISION Topics: discrete convolution • Pre-activations from channel x i into feature map y j can be computed by: ~ ‣ getting the convolution kernel where k ij = W ij from the connection matrix W ij ‣ applying the convolution x i * k ij • This is equivalent to computing the discrete correlation 
 of x i with W ij

  23. 17 COMPUTER VISION Topics: discrete convolution ~ 0% 0.5% • Simple illustration: x i * k ij where W ij = W ij 0.5% 0% W 0.5% 0% 0% 0% 255% 0% 0% 0% 128% 128% 0% 0.5% 0% 0% 0% 255% 0% 0% 0% 128% 128% 0% 0% 0% 255% 0% 0% 0% 255% 0% 0% 0% 255% 0% 0% 0% 255% 0% 0% 0% 255% 0% 0% 0% 0% x i * k ij x i X W %%%%% %%%%%

  24. 18 COMPUTER VISION Topics: discrete convolution • With a non-linearity, we get a detector of a feature at any position in the image 0% 0% 255% 0% 0% 0.02% 0.19% 0.19% 0.02% 0% 128% 128% 0% 0% 0% 255% 0% 0% 0.02% 0.19% 0.19% 0.02% 0% 128% 128% 0% 0% 0% 255% 0% 0% 0.02% 0.75% 0.02% 0.02% 0% 255% 0% 0% 0% 255% 0% 0% 0% 0.75% 0.02% 0.02% 0.02% 255% 0% 0% 0% 255% 0% 0% 0% 0% sigm(0.02 x i * k ij - 4) x i Logis6c (%(%%%%%%%%%%%%%n%200%)%/%50%)%

  25. 19 COMPUTER VISION Topics: discrete convolution • Can use ‘‘zero padding’’ to allow going over the borders ( * ) 0% 0% 255% 0% 0% 0% 128% 128% 0% 0% 0% 255% 0% 0% 0% 128% 128% 0% 0% 0% 255% 0% 0% 0% 255% 0% 0% 0% 255% 0% 0% 0% 255% 0% 0% 0% 255% 0% 0% 0% 0% x i * k ij x i

  26. 19 COMPUTER VISION Topics: discrete convolution • Can use ‘‘zero padding’’ to allow going over the borders ( * ) 0% 255% 0% 255% 0% 0% 0% 0% 0% 0% 0% 0% 0% 255% 0% 255% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 255% 0% 0% 0% 0% 0% 128% 128% 0% 0% 0% 0% 0% 0% 0% 255% 0% 0% 0% 0% 0% 128% 128% 0% 0% 0% 0% 0% 0% 0% 255% 0% 0% 0% 0% 0% 255% 0% 0% 0% 0% 0% 0% 0% 255% 0% 0% 0% 0% 0% 255% 0% 0% 0% 0% 0% 0% 0% 255% 0% 0% 0% 0% 0% 0% x i * k ij x i

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC www.kit.edu KIT The Research University in the Helmholtz Association Neural Network Concept (very) remotely brain inspired computational system

479 views • 25 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
7. Artificial neural networks Introduction to neural networks Despite struggling to understand

7. Artificial neural networks Introduction to neural networks Despite struggling to understand

7. Artificial neural networks Introduction to neural networks Despite struggling to understand intricacies of protein, cell, and network function within the brain, neuroscientists would agree on the following simplistic description of how the

157 views • 11 slides
CS 4803 / 7643: Deep Learning Topics: (Continue) Low-label ML Formulations Zsolt Kira

CS 4803 / 7643: Deep Learning Topics: (Continue) Low-label ML Formulations Zsolt Kira

CS 4803 / 7643: Deep Learning Topics: (Continue) Low-label ML Formulations Zsolt Kira Georgia Tech Administrative Projects! Poster details out on piazza Note: No late days for anything project related! Also note: Keep

864 views • 34 slides
A simple way to create a website with A simple way to create a website with Blogdown Blogdown R

A simple way to create a website with A simple way to create a website with Blogdown Blogdown R

A simple way to create a website with A simple way to create a website with Blogdown Blogdown R Qubec Workshop R Qubec Workshop Mamadou Yauck & Sergio Ewane Ebouele Mamadou Yauck & Sergio Ewane Ebouele 2019-05-13

795 views • 31 slides
OPEN SOURCE HUGO TESO ALLOW ME TO... CYBER-CYBER AIRPLANES! NDA... :( I DO OPEN SAUCE!

OPEN SOURCE HUGO TESO ALLOW ME TO... CYBER-CYBER AIRPLANES! NDA... :( I DO OPEN SAUCE!

BEAUTIFUL OPEN SOURCE HUGO TESO ALLOW ME TO... CYBER-CYBER AIRPLANES! NDA... :( I DO OPEN SAUCE! Hugo Teso INGUMA BOKKEN IAIT DO YOU SEE THE PATTERN? Hugo Teso DO YOU SEE THE PATTERN? CYBER-CYBER? Hugo Teso DO YOU SEE THE

807 views • 51 slides
Creating Websites with Hugo & R Markdown Thomas Lo Russo WHO Research Associate Official

Creating Websites with Hugo & R Markdown Thomas Lo Russo WHO Research Associate Official

Creating Websites with Hugo & R Markdown Thomas Lo Russo WHO Research Associate Official Statistics & Open Government Data AM Canton Zurich Politics & Data-blogger (politan.ch) I @thlorusso TWIST2018 - Hackdays Website

537 views • 20 slides
On the Direction of Innovation Hugo A. Hopenhayn Francesco Squintani EARIE, Milan August 2014

On the Direction of Innovation Hugo A. Hopenhayn Francesco Squintani EARIE, Milan August 2014

On the Direction of Innovation Hugo A. Hopenhayn Francesco Squintani EARIE, Milan August 2014 Hugo A. Hopenhayn, Francesco Squintani On the Direction of Innovation Introduction Most research on patents asks: Is IP sufficient or

962 views • 63 slides
Introd u ction to iterators P YTH ON DATA SC IE N C E TOOL BOX ( PAR T 2 ) H u go Bo w ne -

Introd u ction to iterators P YTH ON DATA SC IE N C E TOOL BOX ( PAR T 2 ) H u go Bo w ne -

Introd u ction to iterators P YTH ON DATA SC IE N C E TOOL BOX ( PAR T 2 ) H u go Bo w ne - Anderson Data Scientist at DataCamp Iterating w ith a for loop We can iterate o v er a list u sing a for loop employees = ['Nick', 'Lore', 'Hugo'] for

379 views • 25 slides
sttde : a time-depending and post- estimation within time-interval command Hugo Sjqvist

sttde : a time-depending and post- estimation within time-interval command Hugo Sjqvist

sttde : a time-depending and post- estimation within time-interval command Hugo Sjqvist hugo.sjoqvist@ki.se Nicola Orsini nicola.orsini@ki.se Exposure data of income (Year_1990-Year_2015) collected over 25 years, with record of the death

401 views • 15 slides
An Education, Research and Innovation Ecosystem 2 Carnegie Mellon Portugal Program The

An Education, Research and Innovation Ecosystem 2 Carnegie Mellon Portugal Program The

An Education, Research and Innovation Ecosystem 2 Carnegie Mellon Portugal Program The Carnegie Mellon Portugal Program is a platform for education, research and innovation that brings together Portuguese universities, research institutions

785 views • 22 slides

More recommend