Neural Networks II Chen Gao Virginia Tech Spring 2019 ECE-5424G / - - PowerPoint PPT Presentation

Neural Networks II

Chen Gao Virginia Tech

Spring 2019

ECE-5424G / CS-5824

Neural Networks

• Origins: Algorithms that try to mimic the brain.

What is this?

A single neuron in the brain

Input Output

Slide credit: Andrew Ng

An artificial neuron: Logistic unit

𝑦 = 𝑦0 𝑦1 𝑦2 𝑦3 𝜄 = 𝜄0 𝜄1 𝜄2 𝜄3

• Sigmoid (logistic) activation function

𝑦1 𝑦2 𝑦3 𝑦0

ℎ𝜄 𝑦 = 1 1 + 𝑓−𝜄⊤𝑦

“Bias unit” “Input” “Output” “Weights” “Parameters”

Slide credit: Andrew Ng

Visualization of weights, bias, activation function

bias b only change the position of the hyperplane

Slide credit: Hugo Larochelle

range determined by g(.)

Activation - sigmoid

• Squashes the neuron’s pre-

activation between 0 and 1

• Always positive
• Bounded
• Strictly increasing

𝑕 𝑦 = 1 1 + 𝑓−𝑦

Slide credit: Hugo Larochelle

Activation - hyperbolic tangent (tanh)

• Squashes the neuron’s pre-

activation between -1 and 1

• Can be positive or negative
• Bounded
• Strictly increasing

𝑕 𝑦 = tanh 𝑦 = 𝑓𝑦 − 𝑓−𝑦 𝑓𝑦 + 𝑓−𝑦

Slide credit: Hugo Larochelle

Activation - rectified linear(relu)

• Bounded below by 0
• always non-negative
• Not upper bounded
• Tends to give neurons with

sparse activities

Slide credit: Hugo Larochelle

𝑕 𝑦 = relu 𝑦 = max 0, 𝑦

Activation - softmax

• For multi-class classification:
• we need multiple outputs (1 output per class)
• we would like to estimate the conditional probability 𝑞 𝑧 = 𝑑 | 𝑦
• We use the softmax activation function at the output:

𝑕 𝑦 = softmax 𝑦 = 𝑓𝑦1 σ𝑑 𝑓𝑦𝑑 … 𝑓𝑦𝑑 σ𝑑 𝑓𝑦𝑑

Slide credit: Hugo Larochelle

Universal approximation theorem

‘‘a single hidden layer neural network with a linear output unit can approximate any continuous function arbitrarily well, given enough hidden units’’ Hornik, 1991

Slide credit: Hugo Larochelle

Neural network – Multilayer

𝑏1

(2)

𝑏2

(2)

𝑏3

(2)

𝑏0

(2)

ℎΘ 𝑦

Layer 1 “Output”

𝑦1 𝑦2 𝑦3 𝑦0

Layer 2 (hidden) Layer 3

Slide credit: Andrew Ng

Neural network

𝑏1

(2)

𝑏2

(2)

𝑏3

(2)

𝑏0

(2)

ℎΘ 𝑦

𝑦1 𝑦2 𝑦3 𝑦0

𝑏𝑗

(𝑘) = “activation” of unit 𝑗 in layer 𝑘

Θ 𝑘 = matrix of weights controlling function mapping from layer 𝑘 to layer 𝑘 + 1 𝑏1

(2) = 𝑕 Θ10 (1)𝑦0 + Θ11 (1)𝑦1 + Θ12 (1)𝑦2 + Θ13 (1)𝑦3

𝑏2

(2) = 𝑕 Θ20 (1)𝑦0 + Θ21 (1)𝑦1 + Θ22 (1)𝑦2 + Θ23 (1)𝑦3

𝑏3

(2) = 𝑕 Θ30 (1)𝑦0 + Θ31 (1)𝑦1 + Θ32 (1)𝑦2 + Θ33 (1)𝑦3

ℎΘ(𝑦) = 𝑕 Θ10

(2)𝑏0 (2) + Θ11 (1)𝑏1 (2) + Θ12 (1)𝑏2 (2) + Θ13 (1)𝑏3 (2)

𝑡

𝑘 unit in layer 𝑘

𝑡

𝑘+1 units in layer 𝑘 + 1

Size of Θ 𝑘 ?

𝑡

𝑘+1 × (𝑡 𝑘 + 1)

Slide credit: Andrew Ng

Neural network

𝑏1

(2)

𝑏2

(2)

𝑏3

(2)

𝑏0

(2)

ℎΘ 𝑦

𝑦1 𝑦2 𝑦3 𝑦0

𝑦 = 𝑦0 𝑦1 𝑦2 𝑦3 z(2) = z1

(2)

z2

(2)

z3

(2)

𝑏1

(2) = 𝑕 Θ10 (1)𝑦0 + Θ11 (1)𝑦1 + Θ12 (1)𝑦2 + Θ13 (1)𝑦3

= 𝑕(z1

(2))

𝑏2

(2) = 𝑕 Θ20 (1)𝑦0 + Θ21 (1)𝑦1 + Θ22 (1)𝑦2 + Θ23 (1)𝑦3

= 𝑕(z2

(2))

𝑏3

(2) = 𝑕 Θ30 (1)𝑦0 + Θ31 (1)𝑦1 + Θ32 (1)𝑦2 + Θ33 (1)𝑦3

= 𝑕(z3

(2))

ℎΘ 𝑦 = 𝑕 Θ10

2 𝑏0 2 + Θ11 1 𝑏1 2 + Θ12 1 𝑏2 2 + Θ13 1 𝑏3 2

= 𝑕(𝑨(3)) “Pre-activation”

Slide credit: Andrew Ng

Why do we need g(.)?

Neural network

𝑏1

(2)

𝑏2

(2)

𝑏3

(2)

𝑏0

(2)

ℎΘ 𝑦

𝑦1 𝑦2 𝑦3 𝑦0

𝑦 = 𝑦0 𝑦1 𝑦2 𝑦3 z(2) = z1

(2)

z2

(2)

z3

(2)

𝑏1

(2) = 𝑕(z1 (2))

𝑏2

(2) = 𝑕(z2 (2))

𝑏3

(2) = 𝑕(z3 (2))

ℎΘ 𝑦 = 𝑕(𝑨(3))

“Pre-activation”

𝑨(2) = Θ(1)𝑦 = Θ(1)𝑏(1) 𝑏(2) = 𝑕(𝑨(2)) Add 𝑏0

(2) = 1

𝑨(3) = Θ(2)𝑏(2) ℎΘ 𝑦 = 𝑏(3) = 𝑕(𝑨(3))

Slide credit: Andrew Ng

Flow graph - Forward propagation

𝑏(2) 𝑨(2) 𝑏(3)

ℎΘ 𝑦

X 𝑋(1) 𝑐(1) 𝑨(3) 𝑋(2) 𝑐(2)

𝑨(2) = Θ(1)𝑦 = Θ(1)𝑏(1) 𝑏(2) = 𝑕(𝑨(2)) Add 𝑏0

(2) = 1

𝑨(3) = Θ(2)𝑏(2) ℎΘ 𝑦 = 𝑏(3) = 𝑕(𝑨(3))

How do we evaluate

• ur prediction?

Cost function

Logistic regression: Neural network:

Slide credit: Andrew Ng

Gradient computation

Need to compute:

Slide credit: Andrew Ng

Gradient computation

Given one training example 𝑦, 𝑧 𝑏(1) = 𝑦 𝑨(2) = Θ(1)𝑏(1) 𝑏(2) = 𝑕(𝑨(2)) (add a0

(2))

𝑨(3) = Θ(2)𝑏(2) 𝑏(3) = 𝑕(𝑨(3)) (add a0

(3))

𝑨(4) = Θ(3)𝑏(3) 𝑏(4) = 𝑕 𝑨 4 = ℎΘ 𝑦

Slide credit: Andrew Ng

Gradient computation: Backpropagation

Intuition: 𝜀

𝑘 (𝑚) = “error” of node 𝑘 in layer 𝑚

For each output unit (layer L = 4)

Slide credit: Andrew Ng

𝜀(4) = 𝑏(4) − 𝑧 𝜀(3) = 𝜀(4) 𝜖𝜀(4)

𝜖𝑨(3) = 𝜀(4) 𝜖𝜀(4) 𝜖𝑏(4) 𝜖𝑏(4) 𝜖𝑨(4) 𝜖𝑨(4) 𝜖𝑏(3) 𝜖𝑏(3) 𝜖𝑨(3)

= 1 *Θ 3 𝑈𝜀(4) .∗ 𝑕′ 𝑨 4 .∗ 𝑕′(𝑨(3))

𝑨(3) = Θ(2)𝑏(2) 𝑏(3) = 𝑕(𝑨(3)) 𝑨(4) = Θ(3)𝑏(3) 𝑏(4) = 𝑕 𝑨 4

Backpropagation algorithm

Training set 𝑦(1), 𝑧(1) … 𝑦(𝑛), 𝑧(𝑛) Set Θ(1) = 0 For 𝑗 = 1 to 𝑛 Set 𝑏(1) = 𝑦 Perform forward propagation to compute 𝑏(𝑚) for 𝑚 = 2. . 𝑀 use 𝑧(𝑗) to compute 𝜀(𝑀) = 𝑏(𝑀) − 𝑧(𝑗) Compute 𝜀(𝑀−1), 𝜀(𝑀−2) … 𝜀(2) Θ(𝑚) = Θ(𝑚) − 𝑏(𝑚)𝜀(𝑚+1)

Slide credit: Andrew Ng

Activation - sigmoid

• Partial derivative

𝑕 𝑦 = 1 1 + 𝑓−𝑦

Slide credit: Hugo Larochelle

𝑕′ 𝑦 = 𝑕 𝑦 1 − 𝑕 𝑦

Activation - hyperbolic tangent (tanh)

𝑕 𝑦 = tanh 𝑦 = 𝑓𝑦 − 𝑓−𝑦 𝑓𝑦 + 𝑓−𝑦

Slide credit: Hugo Larochelle

• Partial derivative

𝑕′ 𝑦 = 1 − 𝑕 𝑦 2

Activation - rectified linear(relu)

Slide credit: Hugo Larochelle

𝑕 𝑦 = relu 𝑦 = max 0, 𝑦

• Partial derivative

𝑕′ 𝑦 = 1𝑦 > 0

Initialization

• For bias
• Initialize all to 0
• For weights
• Can’t initialize all weights to the same value
• we can show that all hidden units in a layer will always behave the same
• need to break symmetry
• Recipe: U[-b, b]
• the idea is to sample around 0 but break symmetry

Slide credit: Hugo Larochelle

Putting it together

Pick a network architecture

Slide credit: Hugo Larochelle

• No. of input units: Dimension of features
• No. output units: Number of classes
• Reasonable default: 1 hidden layer, or if >1 hidden layer, have same no.
• f hidden units in every layer (usually the more the better)
• Grid search

Putting it together

Early stopping

Slide credit: Hugo Larochelle

• Use a validation set performance to select the best configuration
• To select the number of epochs, stop training when validation set error

increases

Other tricks of the trade

Slide credit: Hugo Larochelle

• Normalizing your (real-valued) data
• Decaying the learning rate
• as we get closer to the optimum, makes sense to take smaller update steps
• mini-batch
• can give a more accurate estimate of the risk gradient
• Momentum
• can use an exponential average of previous gradients

Dropout

Slide credit: Hugo Larochelle

• Idea: «cripple» neural network by removing hidden units
• each hidden unit is set to 0 with probability 0.5
• hidden units cannot co-adapt to other units
• hidden units must be more generally useful