Lecture 1: Feedforward Princeton University COS 495 Instructor: - - PowerPoint PPT Presentation

Deep Learning Basics Lecture 1: Feedforward

Princeton University COS 495 Instructor: Yingyu Liang

Motivation I: representation learning

Machine learning 1-2-3

• Collect data and extract features
• Build model: choose hypothesis class 𝓘 and loss function 𝑚
• Optimization: minimize the empirical loss

Features

Color Histogram

Red Green Blue

Extract features

𝑦 𝑧 = 𝑥𝑈𝜚 𝑦

build hypothesis

Features: part of the model

𝑧 = 𝑥𝑈𝜚 𝑦

build hypothesis

Linear model Nonlinear model

Example: Polynomial kernel SVM

𝑦1 𝑦2 𝑧 = sign(𝑥𝑈𝜚(𝑦) + 𝑐) Fixed 𝜚 𝑦

Motivation: representation learning

• Why don’t we also learn 𝜚 𝑦 ?

Learn 𝜚 𝑦

𝑦 𝑧 = 𝑥𝑈𝜚 𝑦

Learn 𝑥

𝜚 𝑦

Feedforward networks

• View each dimension of 𝜚 𝑦 as something to be learned

𝑦

𝜚 𝑦

𝑧 = 𝑥𝑈𝜚 𝑦

… …

Feedforward networks

• Linear functions 𝜚𝑗 𝑦 = 𝜄𝑗

𝑈𝑦 don’t work: need some nonlinearity

𝑦

𝜚 𝑦

𝑧 = 𝑥𝑈𝜚 𝑦

… …

Feedforward networks

• Typically, set 𝜚𝑗 𝑦 = 𝑠(𝜄𝑗

𝑈𝑦) where 𝑠(⋅) is some nonlinear function

𝑦

𝜚 𝑦

𝑧 = 𝑥𝑈𝜚 𝑦

… …

Feedforward deep networks

• What if we go deeper?

… …

… … … …

𝑦

ℎ1 ℎ2 ℎ𝑀 𝑧

Figure from Deep learning, by Goodfellow, Bengio, Courville. Dark boxes are things to be learned.

Motivation II: neurons

Motivation: neurons

Figure from Wikipedia

Motivation: abstract neuron model

• Neuron activated when the correlation

between the input and a pattern 𝜄 exceeds some threshold 𝑐

• 𝑧 = threshold(𝜄𝑈𝑦 − 𝑐)
• r 𝑧 = 𝑠(𝜄𝑈𝑦 − 𝑐)
• 𝑠(⋅) called activation function

𝑧 𝑦1 𝑦2 𝑦𝑒

Motivation: artificial neural networks

Motivation: artificial neural networks

• Put into layers: feedforward deep networks

… …

… … … …

𝑦

ℎ1 ℎ2 ℎ𝑀 𝑧

Components in Feedforward networks

Components

• Representations:
• Input
• Hidden variables
• Layers/weights:
• Hidden layers
• Output layer

Components

… …

… … … …

Hidden variables ℎ1 ℎ2 ℎ𝑀 𝑧 Input 𝑦 First layer Output layer

Input

• Represented as a vector
• Sometimes require some

preprocessing, e.g.,

• Subtract mean
• Normalize to [-1,1]

Expand

Output layers

• Regression: 𝑧 = 𝑥𝑈ℎ + 𝑐
• Linear units: no nonlinearity

ℎ 𝑧 Output layer

Output layers

• Multi-dimensional regression: 𝑧 = 𝑋𝑈ℎ + 𝑐
• Linear units: no nonlinearity

ℎ 𝑧 Output layer

Output layers

• Binary classification: 𝑧 = 𝜏(𝑥𝑈ℎ + 𝑐)
• Corresponds to using logistic regression on ℎ

ℎ 𝑧 Output layer

Output layers

• Multi-class classification:
• 𝑧 = softmax 𝑨 where 𝑨 = 𝑋𝑈ℎ + 𝑐
• Corresponds to using multi-class

logistic regression on ℎ

ℎ 𝑧 Output layer 𝑨

Hidden layers

• Neuron take weighted linear

combination of the previous layer

• So can think of outputting one

value for the next layer

… …

ℎ𝑗 ℎ𝑗+1

Hidden layers

• 𝑧 = 𝑠(𝑥𝑈𝑦 + 𝑐)
• Typical activation function 𝑠
• Threshold t 𝑨 = 𝕁[𝑨 ≥ 0]
• Sigmoid 𝜏 𝑨 = 1/ 1 + exp(−𝑨)
• Tanh tanh 𝑨 = 2𝜏 2𝑨 − 1

𝑧 𝑦 𝑠(⋅)

Hidden layers

• Problem: saturation

𝑧 𝑦 𝑠(⋅)

Figure borrowed from Pattern Recognition and Machine Learning, Bishop

Too small gradient

Hidden layers

• Activation function ReLU (rectified linear unit)
• ReLU 𝑨 = max{𝑨, 0}

Figure from Deep learning, by Goodfellow, Bengio, Courville.

Hidden layers

• Activation function ReLU (rectified linear unit)
• ReLU 𝑨 = max{𝑨, 0}

Gradient 0 Gradient 1

Hidden layers

• Generalizations of ReLU gReLU 𝑨 = max 𝑨, 0 + 𝛽 min{𝑨, 0}
• Leaky-ReLU 𝑨 = max{𝑨, 0} + 0.01 min{𝑨, 0}
• Parametric-ReLU 𝑨 : 𝛽 learnable

𝑨 gReLU 𝑨