Introduction to Deep Learning A. G. Schwing & S. Fidler - - PowerPoint PPT Presentation

▶
introduction to deep learning
SMART_READER_LITE
LIVE PREVIEW

Introduction to Deep Learning A. G. Schwing & S. Fidler - - PowerPoint PPT Presentation

Jun 05, 2023 109 likes •513 views

Introduction to Deep Learning A. G. Schwing & S. Fidler University of Toronto, 2014 A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding 2014 1 / 35 Outline Universality of Neural Networks 1 Learning Neural

slide-1
SLIDE 1

Introduction to Deep Learning

  • A. G. Schwing & S. Fidler

University of Toronto, 2014

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 1 / 35

slide-2
SLIDE 2

Outline

1

Universality of Neural Networks

2

Learning Neural Networks

3

Deep Learning

4

Applications

5

References

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 2 / 35

slide-3
SLIDE 3

What are neural networks? Let’s ask

  • Biological
  • Computational

Input #1 Input #2 Input #3 Input #4 Output Hidden layer Input layer Output layer

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 3 / 35

slide-4
SLIDE 4

What are neural networks? ...Neural networks (NNs) are computational models inspired by biological neural networks [...] and are used to estimate or approximate functions... [Wikipedia]

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 4 / 35

slide-5
SLIDE 5

What are neural networks? The people behind Origins: Traced back to threshold logic [W. McCulloch and W. Pitts, 1943] Perceptron [F . Rosenblatt, 1958] More recently:

  • G. Hinton (UofT)
  • Y. LeCun (NYU)
  • A. Ng (Stanford)
  • Y. Bengio (University of Montreal)
  • J. Schmidhuber (IDSIA, Switzerland)
  • R. Salakhutdinov (UofT)
  • H. Lee (University of Michigan)
  • R. Fergus (NYU)

... (and many more having made significant contributions; apologies for not being able to mention everyone and all the students behind)

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 5 / 35

slide-6
SLIDE 6

What are neural networks? Use cases Classification Playing video games Captcha Neural Turing Machine (e.g., learn how to sort) Alex Graves

http://www.technologyreview.com/view/532156/googles-secretive-deepmind-startup-unveils-a-neural-turing-machine/

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 6 / 35

slide-7
SLIDE 7

What are neural networks? Example: input x parameters w1, w2, b x ∈ R h1 b ∈ R f w1 w2

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 7 / 35

slide-8
SLIDE 8

How to compute the function? Forward propagation/pass, inference, prediction: Given input x and parameters w, b Compute latent variables/intermediate results in a feed-forward manner Until we obtain output function f x ∈ R h1 b ∈ R f w1 w2

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 8 / 35

slide-9
SLIDE 9

How to compute the function? Forward propagation/pass, inference, prediction: Given input x and parameters w, b Compute latent variables/intermediate results in a feed-forward manner Until we obtain output function f

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 8 / 35

slide-10
SLIDE 10

How to compute the function? Example: input x, parameters w1, w2, b x ∈ R h1 b ∈ R f w1 w2 h1 = σ(w1 · x + b) f = w2 · h1 Sigmoid function: σ(z) = 1/(1 + exp(−z)) x = ln 2, b = ln 3, w1 = 2, w2 = 2 h1 =? f =?

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 9 / 35

slide-11
SLIDE 11

How to compute the function? Given parameters, what is f for x = 0, x = 1, x = 2, ... f = w2σ(w1 · x + b)

−5 5 0.5 1 1.5 2 x f

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 10 / 35

slide-12
SLIDE 12

Let’s mess with parameters: x ∈ R h1 b ∈ R f w1 w2 h1 = σ(w1 · x + b) f = w2 · h1 σ(z) = 1/(1 + exp(−z)) w1 = 1.0 b = 0

−5 5 0.2 0.4 0.6 0.8 1 x f b = −2 b = 0 b = 2 −5 5 0.2 0.4 0.6 0.8 1 x f w1 = 0 w1 = 0.5 w1 = 1.0 w1 = 100

Keep in mind the step function.

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 11 / 35

slide-13
SLIDE 13

How to use Neural Networks for binary classification? Feature/Measurement: x Output: How likely is the input to be a cat?

−5 5 0.2 0.4 0.6 0.8 1 x y

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 12 / 35

slide-14
SLIDE 14

How to use Neural Networks for binary classification? Feature/Measurement: x Output: How likely is the input to be a cat?

−5 5 0.2 0.4 0.6 0.8 1 x f

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 12 / 35

slide-15
SLIDE 15

How to use Neural Networks for binary classification? Feature/Measurement: x Output: How likely is the input to be a cat?

−5 5 0.2 0.4 0.6 0.8 1 x f

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 12 / 35

slide-16
SLIDE 16

How to use Neural Networks for binary classification? Feature/Measurement: x Output: How likely is the input to be a cat?

−5 5 0.2 0.4 0.6 0.8 1 x f

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 12 / 35

slide-17
SLIDE 17

How to use Neural Networks for binary classification? Shifted feature/measurement: x Output: How likely is the input to be a cat? Previous features Shifted features

−5 5 0.2 0.4 0.6 0.8 1 x f −5 5 0.2 0.4 0.6 0.8 1 x f

Learning/Training means finding the right parameters.

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 13 / 35

slide-18
SLIDE 18

So far we are able to scale and translate sigmoids. How well can we approximate an arbitrary function? With the simple model we are obviously not going very far. Features are good Features are noisy Simple classifier More complex classifier

−5 5 0.2 0.4 0.6 0.8 1 x f −5 5 0.2 0.4 0.6 0.8 1 x f

How can we generalize?

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 14 / 35

slide-19
SLIDE 19

Let’s use more hidden variables:

x ∈ R h1 b1 h2 b2 f w1 w2 w3 w4

h1 = σ(w1 · x + b1) h2 = σ(w3 · x + b2) f = w2 · h1 + w4 · h2 Combining two step functions gives a bump.

−5 5 1 1.2 1.4 1.6 1.8 2 x f

w1 = −100, b1 = 40, w3 = 100, b2 = 60, w2 = 1, w4 = 1

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 15 / 35

slide-20
SLIDE 20

So let’s simplify:

x ∈ R h1 b1 h2 b2 f w1 w2 w3 w4 f Bump(x1, x2, h)

We simplify a pair of hidden nodes to a “bump” function: Starts at x1 Ends at x2 Has height h

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 16 / 35

slide-21
SLIDE 21

Now we can represent “bumps” very well. How can we generalize?

f Bump(0.0, 0.2, h1) Bump(0.2, 0.4, h2) Bump(0.4, 0.6, h3) Bump(0.6, 0.8, h4) Bump(0.8, 1.0, h5) 0.5 1 −0.5 0.5 1 1.5 x f Target Approximation

More bumps gives more accurate approximation. Corresponds to a single layer network.

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 17 / 35

slide-22
SLIDE 22

Universality: theoretically we can approximate an arbitrary function So we can learn a really complex cat classifier Where is the catch?

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 18 / 35

slide-23
SLIDE 23

Universality: theoretically we can approximate an arbitrary function So we can learn a really complex cat classifier Where is the catch? Complexity, we might need quite a few hidden units Overfitting, memorize the training data

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 18 / 35

slide-24
SLIDE 24

Generalizations are possible to include more input dimensions capture more output dimensions employ multiple layers for more efficient representations See ‘http://neuralnetworksanddeeplearning.com/chap4.html’ for a great read!

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 19 / 35

slide-25
SLIDE 25

How do we find the parameters to obtain a good approximation? How do we tell a computer to do that? Intuitive explanation: Compute approximation error at the output Propagate error back by computing individual contributions of parameters to error

[Fig. from H. Lee]

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 20 / 35

slide-26
SLIDE 26

Intuitive example: Target function: 5x2 Approximation: fw(x) Domain of interest: x ∈ [0, 1] Error: e(w) = 1 (5x2 − fw(x))2dx Program of interest: min

w e(w) = min w

1 (5x2 − fw(x))2dx

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 21 / 35

slide-27
SLIDE 27

Chain rule is important: w1, w2, x ∈ R Assume e(w1, w2) = f(w2, h(w1, x)) Derivatives are: ∂e(w1, w2) ∂w2 = ∂f(w2, h(w1, x)) ∂w2 ∂e(w1, w2) ∂w1 = ∂f(w2, h(w1, x)) ∂w1 = ∂f ∂h · ∂h ∂w1 Chain rule

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 22 / 35

slide-28
SLIDE 28

Back propagation does not work well for deep networks: Diffusion of gradient signal (multiplication of many small numbers) Attractivity of many local minima (random initialization is very far from good points) Requires a lot of training samples Need for significant computational power Solution: 2 step approach Greedy layerwise pre-training Perform full fine tuning at the end

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 23 / 35

slide-29
SLIDE 29

Why go deep? Representation efficiency (fewer computational units for the same function) Hierarchical representation (non-local generalization) Combinatorial sharing (re-use of earlier computation) Works very well

[Fig. from H. Lee]

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 24 / 35

slide-30
SLIDE 30

To obtain more flexibility/non-linearity we use additional function prototypes: Sigmoid Rectified linear unit (ReLU) Pooling Dropout Convolutions

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 25 / 35

slide-31
SLIDE 31

Convolutions What do the numbers mean? See Sanja’s lecture 14 for the answers...

[Fig. adapted from A. Krizhevsky]

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 26 / 35

slide-32
SLIDE 32

Max Pooling What is happening here?

[Fig. adapted from A. Krizhevsky]

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 27 / 35

slide-33
SLIDE 33

Rectified Linear Unit (ReLU) Drop information if smaller than zero Fixes the problem of vanishing gradients to some degree Dropout Drop information at random Kind of a regularization, enforcing redundancy

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 28 / 35

slide-34
SLIDE 34

A famous deep learning network called “AlexNet.” The network won the ImageNet competition in 2012. How many parameters? Given an image, what is happening? Inference Time: about 2ms per image when processing many images in parallel Training Time: forever (maybe 2-3 weeks)

[Fig. adapted from A. Krizhevsky]

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 29 / 35

slide-35
SLIDE 35

Demo

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 30 / 35

slide-36
SLIDE 36

Neural networks have been used for many applications: Classification and Recognition in Computer Vision Text Parsing in Natural Language Processing Playing Video Games Stock Market Prediction Captcha Demos: Russ website Antonio Places website

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 31 / 35

slide-37
SLIDE 37

Classification in Computer Vision: ImageNet Challenge

http://deeplearning.cs.toronto.edu/

Since it’s the end of the semester, let’s find the beach...

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 32 / 35

slide-38
SLIDE 38

Classification in Computer Vision: ImageNet Challenge

http://deeplearning.cs.toronto.edu/

A place to maybe prepare for exams...

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 33 / 35

slide-39
SLIDE 39

Links: Tutorials: http://deeplearning.net/tutorial/deeplearning.pdf Toronto Demo by Russ and students: http://deeplearning.cs.toronto.edu/ MIT Demo by Antonio and students: http://places.csail.mit.edu/demo.html Honglak Lee: http://deeplearningworkshopnips2010.files.wordpress.com/2010/09/nips10- workshop-tutorial-final.pdf Yann LeCun: http://www.cs.nyu.edu/ yann/talks/lecun-ranzato-icml2013.pdf Richard Socher: http://lxmls.it.pt/2014/socher-lxmls.pdf

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 34 / 35

slide-40
SLIDE 40

Videos: Video games: https://www.youtube.com/watch?v=mARt-xPablE Captcha: http://singularityhub.com/2013/10/29/tiny-ai-startup- vicarious-says-its-solved-captcha/ https://www.youtube.com/watch?v=lge-dl2JUAM#t=27 Stock exchange: http://cs.stanford.edu/people/eroberts/courses/soco/projects/neural- networks/Applications/stocks.html

  • A. G. Schwing & S. Fidler (UofT)

CSC420: Intro to Image Understanding 2014 35 / 35