Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 - - PowerPoint PPT Presentation

Optimization and Backpropagation

1 I2DL: Prof. Niessner, Prof. Leal-Taixé

Lecture 3 Recap

2 I2DL: Prof. Niessner, Prof. Leal-Taixé

Neural Network

• Linear score function 𝒈 = 𝑿𝒚

3 I2DL: Prof. Niessner, Prof. Leal-Taixé

On CIFAR-10 On ImageNet

Credit: Li/Karpathy/Johnson

Neural Network

• Linear score function 𝒈 = 𝑿𝒚
• Neural network is a nesting of ‘functions’

– 2-layers: 𝒈 = 𝑿𝟑 max(𝟏, 𝑿𝟐𝒚) – 3-layers: 𝒈 = 𝑿𝟒 max(𝟏, 𝑿𝟑 max(𝟏, 𝑿𝟐𝒚)) – 4-layers: 𝒈 = 𝑿𝟓 tanh (𝑿𝟒, max(𝟏, 𝑿𝟑 max(𝟏, 𝑿𝟐𝒚))) – 5-layers: 𝒈 = 𝑿𝟔𝜏(𝑿𝟓 tanh(𝑿𝟒, max(𝟏, 𝑿𝟑 max(𝟏, 𝑿𝟐𝒚)))) – … up to hundreds of layers

4 I2DL: Prof. Niessner, Prof. Leal-Taixé

Neural Network

5 I2DL: Prof. Niessner, Prof. Leal-Taixé

Credit: Li/Karpathy/Johnson

Input layer Hidden layer Output layer

Neural Network

Depth Width

6

Input Layer Hidden Layer 1 Hidden Layer 2 Hidden Layer 3 Output Layer

I2DL: Prof. Niessner, Prof. Leal-Taixé

Activation Functions

7 I2DL: Prof. Niessner, Prof. Leal-Taixé

Sigmoid: 𝜏 𝑦 =

1 (1+𝑓−𝑦)

tanh: tanh 𝑦 ReLU: max 0, 𝑦 Leaky ReLU: max 0.1𝑦, 𝑦 Maxout max 𝑥1

𝑈𝑦 + 𝑐1, 𝑥2 𝑈𝑦 + 𝑐2

ELU f x = ቊ 𝑦 if 𝑦 > 0 α e𝑦 − 1 if 𝑦 ≤ 0 Parametric ReLU: max 𝛽𝑦, 𝑦

Loss Functions

• Measure the goodness of the predictions (or

equivalently, the network's performance)

8 I2DL: Prof. Niessner, Prof. Leal-Taixé

• Regression loss

– L1 loss 𝑀 𝒛, ෝ 𝒛; 𝜾 = 1

𝑜 σ𝑗 𝑜 | 𝑧𝑗 − ෝ

𝑧𝑗| 1 – MSE loss 𝑀 𝒛, ෝ 𝒛; 𝜾 = 1

𝑜 σ𝑗 𝑜 | 𝑧𝑗 − ෝ

𝑧𝑗| 2

2

• Classification loss (for multi-class classification)

– Cross Entropy loss E 𝒛, ෝ 𝒛; 𝜾 = − σ𝑗=1

𝑜

σ𝑙=1

𝑙

(𝑧𝑗𝑙 ∙ log ො 𝑧𝑗𝑙)

Computational Graphs

• Neural network is a computational graph

– It has compute nodes – It has edges that connect nodes – It is directional – It is organized in ‘layers’

9 I2DL: Prof. Niessner, Prof. Leal-Taixé

Backprop

10 I2DL: Prof. Niessner, Prof. Leal-Taixé

The Importance of Gradients

• Our optimization schemes are based on computing

gradients

• One can compute gradients analytically but what if
• ur function is too complex?
• Break down gradient computation

11 I2DL: Prof. Niessner, Prof. Leal-Taixé

Backpropagation

𝛼𝜾𝑀 𝜾

Rumelhart 1986

Backprop: Forward Pass

• 𝑔 𝑦, 𝑧, 𝑨 = 𝑦 + 𝑧 ⋅ 𝑨

12 I2DL: Prof. Niessner, Prof. Leal-Taixé

Initialization 𝑦 = 1, 𝑧 = −3, 𝑨 = 4 mult sum 𝑔 = −8 1 −3 4 𝑒 = −2 1 −3 4

Backprop: Backward Pass

13 I2DL: Prof. Niessner, Prof. Leal-Taixé

with 𝑦 = 1, 𝑧 = −3, 𝑨 = 4

mult sum 𝑔 = −8 1 −3 4 𝑒 = −2 1 −3 4

𝑔 𝑦, 𝑧, 𝑨 = 𝑦 + 𝑧 ⋅ 𝑨 𝑒 = 𝑦 + 𝑧

𝜖𝑒 𝜖𝑦 = 1, 𝜖𝑒 𝜖𝑧 = 1

𝑔 = 𝑒 ⋅ 𝑨

𝜖𝑔 𝜖𝑒 = 𝑨, 𝜖𝑔 𝜖𝑨 = 𝑒

What is

𝜖𝑔 𝜖𝑦 , 𝜖𝑔 𝜖𝑧 , 𝜖𝑔 𝜖𝑨 ?

Backprop: Backward Pass

14 I2DL: Prof. Niessner, Prof. Leal-Taixé

with 𝑦 = 1, 𝑧 = −3, 𝑨 = 4

mult sum 𝑔 = −8 1 −3 4 𝑒 = −2 1 −3 4

𝑔 𝑦, 𝑧, 𝑨 = 𝑦 + 𝑧 ⋅ 𝑨 𝑒 = 𝑦 + 𝑧

𝜖𝑒 𝜖𝑦 = 1, 𝜖𝑒 𝜖𝑧 = 1

𝑔 = 𝑒 ⋅ 𝑨

𝜖𝑔 𝜖𝑒 = 𝑨, 𝜖𝑔 𝜖𝑨 = 𝑒

What is

𝜖𝑔 𝜖𝑦 , 𝜖𝑔 𝜖𝑧 , 𝜖𝑔 𝜖𝑨 ?

𝜖𝑔 𝜖𝑔 1

Backprop: Backward Pass

15 I2DL: Prof. Niessner, Prof. Leal-Taixé

with 𝑦 = 1, 𝑧 = −3, 𝑨 = 4

mult sum 𝑔 = −8 1 −3 4 𝑒 = −2 1 −3 4

𝑔 𝑦, 𝑧, 𝑨 = 𝑦 + 𝑧 ⋅ 𝑨 𝑒 = 𝑦 + 𝑧

𝜖𝑒 𝜖𝑦 = 1, 𝜖𝑒 𝜖𝑧 = 1

𝑔 = 𝑒 ⋅ 𝑨

𝜖𝑔 𝜖𝑒 = 𝑨, 𝜖𝑔 𝜖𝑨 = 𝑒

What is

𝜖𝑔 𝜖𝑦 , 𝜖𝑔 𝜖𝑧 , 𝜖𝑔 𝜖𝑨 ?

1 𝜖𝑔 𝜖𝑨 −2 −2

Backprop: Backward Pass

16 I2DL: Prof. Niessner, Prof. Leal-Taixé

with 𝑦 = 1, 𝑧 = −3, 𝑨 = 4

mult sum 𝑔 = −8 1 −3 4 𝑒 = −2 1 −3 4

𝑔 𝑦, 𝑧, 𝑨 = 𝑦 + 𝑧 ⋅ 𝑨 𝑒 = 𝑦 + 𝑧

𝜖𝑒 𝜖𝑦 = 1, 𝜖𝑒 𝜖𝑧 = 1

𝑔 = 𝑒 ⋅ 𝑨

𝜖𝑔 𝜖𝑒 = 𝑨, 𝜖𝑔 𝜖𝑨 = 𝑒

What is

𝜖𝑔 𝜖𝑦 , 𝜖𝑔 𝜖𝑧 , 𝜖𝑔 𝜖𝑨 ?

−2 𝜖𝑔 𝜖𝑒 4 1 −2

Backprop: Backward Pass

17 I2DL: Prof. Niessner, Prof. Leal-Taixé

with 𝑦 = 1, 𝑧 = −3, 𝑨 = 4

mult sum 𝑔 = −8 1 −3 4 𝑒 = −2 1 −3 4

𝑔 𝑦, 𝑧, 𝑨 = 𝑦 + 𝑧 ⋅ 𝑨 𝑒 = 𝑦 + 𝑧

𝜖𝑒 𝜖𝑦 = 1, 𝜖𝑒 𝜖𝑧 = 1

𝑔 = 𝑒 ⋅ 𝑨

𝜖𝑔 𝜖𝑒 = 𝑨, 𝜖𝑔 𝜖𝑨 = 𝑒

What is

𝜖𝑔 𝜖𝑦 , 𝜖𝑔 𝜖𝑧 , 𝜖𝑔 𝜖𝑨 ?

1 4 𝜖𝑔 𝜖𝑧 4 𝜖𝑔 𝜖𝑧 = 𝜖𝑔 𝜖𝑒 ⋅ 𝜖𝑒 𝜖𝑧 Chain Rule:

→ 𝜖𝑔 𝜖𝑧 = 4 ⋅ 1 = 4

4 −2

Backprop: Backward Pass

18 I2DL: Prof. Niessner, Prof. Leal-Taixé

with 𝑦 = 1, 𝑧 = −3, 𝑨 = 4

mult sum 𝑔 = −8 1 −3 4 𝑒 = −2 1 −3 4

𝑔 𝑦, 𝑧, 𝑨 = 𝑦 + 𝑧 ⋅ 𝑨 𝑒 = 𝑦 + 𝑧

𝜖𝑒 𝜖𝑦 = 1, 𝜖𝑒 𝜖𝑧 = 1

𝑔 = 𝑒 ⋅ 𝑨

𝜖𝑔 𝜖𝑒 = 𝑨, 𝜖𝑔 𝜖𝑨 = 𝑒

What is

𝜖𝑔 𝜖𝑦 , 𝜖𝑔 𝜖𝑧 , 𝜖𝑔 𝜖𝑨 ?

1 −2 4 4 −2 𝜖𝑔 𝜖𝑦 = 𝜖𝑔 𝜖𝑒 ⋅ 𝜖𝑒 𝜖𝑦 Chain Rule:

→ 𝜖𝑔 𝜖𝑦 = 4 ⋅ 1 = 4

𝜖𝑔 𝜖𝑦 4 4 4

Compute Graphs -> Neural Networks

• 𝑦𝑙 input variables
• 𝑥𝑚,𝑛,𝑜 network weights (note 3 indices)

– 𝑚 which layer – 𝑛 which neuron in layer – 𝑜 which weight in neuron

• ො

𝑧𝑗 computed output (𝑗 output dim; 𝑜𝑝𝑣𝑢)

• 𝑧𝑗 ground truth targets
• 𝑀 loss function

19 I2DL: Prof. Niessner, Prof. Leal-Taixé

Compute Graphs -> Neural Networks

20 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦0 𝑦1 ො 𝑧0 𝑧0 Input layer Output layer e.g., class label/ regression target 𝑦0 𝑦1 ∗ 𝑥1 ∗ 𝑥0 −𝑧0 + 𝑦∗𝑦 Input Weights (unknowns!) L2 Loss function Loss/ cost

Compute Graphs -> Neural Networks

21 I2DL: Prof. Niessner, Prof. Leal-Taixé

Input layer Output layer e.g., class label/ regression target 𝑦0 𝑦1 ∗ 𝑥1 ∗ 𝑥0 + Input Weights (unknowns!) L2 Loss function Loss/ cost We want to compute gradients w.r.t. all weights 𝑿 max(0, 𝑦) ReLU Activation

(btw. I’m not arguing this is the right choice here)

𝑦∗𝑦 −𝑧0

𝑦0 𝑦1 ො 𝑧0 𝑧0

Compute Graphs -> Neural Networks

22 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦0 𝑦1 ො 𝑧0 𝑧0 Input layer Output layer ො 𝑧1 ො 𝑧2 𝑧1 𝑧2 𝑦0 𝑦1 ∗ 𝑥0,0 + Loss/ cost + Loss/ cost ∗ 𝑥0,1 ∗ 𝑥1,0 ∗ 𝑥1,1 + Loss/ cost ∗ 𝑥2,0 ∗ 𝑥2,1 We want to compute gradients w.r.t. all weights 𝑿 𝑦∗𝑦 𝑦∗𝑦 𝑦∗𝑦 −𝑧0 −𝑧0 −𝑧0

Compute Graphs -> Neural Networks

23 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦0 𝑦𝑙 ො 𝑧0 𝑧0 Input layer Output layer ො 𝑧1 𝑧1 … ො 𝑧𝑗 = 𝐵(𝑐𝑗 + ෍

𝑙

𝑦𝑙𝑥𝑗,𝑙) 𝑀 = ෍

𝑗

𝑀𝑗 𝑀𝑗 = ො 𝑧𝑗 − 𝑧𝑗 2 We want to compute gradients w.r.t. all weights 𝑿 AND all biases 𝒄 Activation function bias 𝜖𝑀𝑗 𝜖𝑥𝑗,𝑙 = 𝜖𝑀𝑗 𝜖 ො 𝑧𝑗 ⋅ 𝜖 ො 𝑧𝑗 𝜖𝑥𝑗,𝑙 ⟶ use chain rule to compute partials Goal: We want to compute gradients of the loss function 𝑀 w.r.t. all weights 𝑿 𝑀: sum over loss per sample, e.g. L2 loss ⟶ simply sum up squares:

NNs as Computational Graphs

• We can express any kind of functions in a

computational graph, e.g. 𝑔 𝒙, 𝒚 =

1 1+𝑓− 𝑐+𝑥0𝑦0+𝑥1𝑦1

24 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑥1 𝑦1 ∗ 𝑥0 𝑦0 ∗ 𝑐 + +

∙ −1

+1

1 ∙

Sigmoid function 𝜏 𝑦 = 1 1 + 𝑓−𝑦

exp(∙)

NNs as Computational Graphs

• 𝑔 𝒙, 𝒚 =

1 1+𝑓− 𝑐+𝑥0𝑦0+𝑥1𝑦1

25 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑥1 𝑦1 ∗ 𝑥0 𝑦0 ∗ 𝑐 + +

∙ −1

+1

1 ∙

exp(∙)

2 −1 −3 −2 −3 −2 6 4 1 −1 0.37 1.37 0.73

NNs as Computational Graphs

• 𝑔 𝒙, 𝒚 =

1 1+𝑓− 𝑐+𝑥0𝑦0+𝑥1𝑦1

26 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑥1 𝑦1 ∗ 𝑥0 𝑦0 ∗ 𝑐 + +

∙ −1

+1

1 ∙

exp(∙)

2 −1 −3 −2 −3 −2 6 4 1 −1 0.37 1.37 0.73 1 −0.53 𝑕 𝑦 = 1

𝑦

⇒

𝜖𝑕 𝜖𝑦 = − 1 𝑦2

𝑕𝛽 𝑦 = 𝛽 + 𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 1

𝑕 𝑦 = 𝑓𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 𝑓𝑦

𝑕𝛽 𝑦 = 𝛽𝑦 ⇒

𝜖𝑕 𝜖𝑦 = 𝛽

1 ∙ −

1 1.372 = −0.53

NNs as Computational Graphs

• 𝑔 𝒙, 𝒚 =

1 1+𝑓− 𝑐+𝑥0𝑦0+𝑥1𝑦1

27 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑥1 𝑦1 ∗ 𝑥0 𝑦0 ∗ 𝑐 + +

∙ −1

+1

1 ∙

exp(∙)

2 −1 −3 −2 −3 −2 6 4 1 −1 0.37 1.37 0.73 1 −0.53 −0.53 𝑕 𝑦 = 1

𝑦

⇒

𝜖𝑕 𝜖𝑦 = − 1 𝑦2

𝑕𝛽 𝑦 = 𝛽 + 𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 1

𝑕 𝑦 = 𝑓𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 𝑓𝑦

𝑕𝛽 𝑦 = 𝛽𝑦 ⇒

𝜖𝑕 𝜖𝑦 = 𝛽

−0.53 ∙ 1 = −0.53

NNs as Computational Graphs

• 𝑔 𝒙, 𝒚 =

1 1+𝑓− 𝑐+𝑥0𝑦0+𝑥1𝑦1

28 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑥1 𝑦1 ∗ 𝑥0 𝑦0 ∗ 𝑐 + +

∙ −1

+1

1 ∙

exp(∙)

2 −1 −3 −2 −3 −2 6 4 1 −1 0.37 1.37 0.73 1 −0.53 −0.53 −0.2 𝑕 𝑦 = 1

𝑦

⇒

𝜖𝑕 𝜖𝑦 = − 1 𝑦2

𝑕𝛽 𝑦 = 𝛽 + 𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 1

𝑕 𝑦 = 𝑓𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 𝑓𝑦

𝑕𝛽 𝑦 = 𝛽𝑦 ⇒

𝜖𝑕 𝜖𝑦 = 𝛽

−0.2 −0.53 ∙ e−1 = −0.2

NNs as Computational Graphs

• 𝑔 𝒙, 𝒚 =

1 1+𝑓− 𝑐+𝑥0𝑦0+𝑥1𝑦1

29 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑥1 𝑦1 ∗ 𝑥0 𝑦0 ∗ 𝑐 + +

∙ −1

+1

1 ∙

exp(∙)

2 −1 −3 −2 −3 −2 6 4 1 −1 0.37 1.37 0.73 1 −0.53 −0.53 −0.2 0.2 𝑕 𝑦 = 1

𝑦

⇒

𝜖𝑕 𝜖𝑦 = − 1 𝑦2

𝑕𝛽 𝑦 = 𝛽 + 𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 1

𝑕 𝑦 = 𝑓𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 𝑓𝑦

𝑕𝛽 𝑦 = 𝛽𝑦 ⇒

𝜖𝑕 𝜖𝑦 = 𝛽

−0.2 ∙ −1 = 0.2

NNs as Computational Graphs

• 𝑔 𝒙, 𝒚 =

1 1+𝑓− 𝑐+𝑥0𝑦0+𝑥1𝑦1

30 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑥1 𝑦1 ∗ 𝑥0 𝑦0 ∗ 𝑐 + +

∙ −1

+1

1 ∙

exp(∙)

2 −1 −3 −2 −3 −2 6 4 1 −1 0.37 1.37 0.73 1 −0.53 −0.53 −0.2 0.2 0.2 0.2 0.4 −0.6 −0.2 −0.4 0.2 0.2 𝑕 𝑦 = 1

𝑦

⇒

𝜖𝑕 𝜖𝑦 = − 1 𝑦2

𝑕𝛽 𝑦 = 𝛽 + 𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 1

𝑕 𝑦 = 𝑓𝑦 ⇒ 𝜖𝑕

𝜖𝑦 = 𝑓𝑦

𝑕𝛽 𝑦 = 𝛽𝑦 ⇒

𝜖𝑕 𝜖𝑦 = 𝛽

Gradient Descent

31 I2DL: Prof. Niessner, Prof. Leal-Taixé

Gradient Descent

Optimum Initialization

𝒚∗ = arg min 𝑔(𝒚)

32 I2DL: Prof. Niessner, Prof. Leal-Taixé

Gradient Descent

• From derivative to gradient
• Gradient steps in direction of negative gradient

33 I2DL: Prof. Niessner, Prof. Leal-Taixé

Direction of greatest increase

• f the function

ⅆ𝑔 𝑦 ⅆ𝑦 𝛼

𝑦𝑔 𝑦

𝛼

𝑦𝑔(𝑦)

𝑦 Learning rate 𝑦′ = 𝑦 − 𝛽𝛼

𝑦𝑔 𝑦

Gradient Descent for Neural Networks

34

Input Layer Hidden Layer 1 Hidden Layer 2 Hidden Layer 3 Output Layer 𝑛 Neurons 𝑚 Layers

I2DL: Prof. Niessner, Prof. Leal-Taixé

Gradient Descent for Neural Networks

𝛼𝑿𝑔 𝒚,𝒛 (𝑿) = 𝜖𝑔 𝜖𝑥0,0,0 … … 𝜖𝑔 𝜖𝑥𝑚,𝑛,𝑜

For a given training pair {𝒚, 𝒛}, we want to update all weights, i.e., we need to compute the derivatives w.r.t. to all weights:

𝑚 Layers 𝑛 Neurons 𝑿′ = 𝑿 − 𝛽𝛼𝑿𝑔 𝒚,𝒛 (𝑿)

Gradient step:

35

Input Layer Hidden Layer 1 Hidden Layer 2 Hidden Layer 3 Output Layer

I2DL: Prof. Niessner, Prof. Leal-Taixé

NNs can Become Quite Complex…

• These graphs can be huge!

[Szegedy et al.,CVPR’15] Going Deeper with Convolutions

36 I2DL: Prof. Niessner, Prof. Leal-Taixé

The Flow of the Gradients

• Many many many many of these nodes form a neural

network

• Each one has its own work to do

37 I2DL: Prof. Niessner, Prof. Leal-Taixé

NEURONS FORWARD AND BACKWARD PASS

The Flow of the Gradients

38 I2DL: Prof. Niessner, Prof. Leal-Taixé

Activations Activation function 𝑨 = 𝑔(𝑦, 𝑧)

𝑦 𝑧

𝜖𝑀 𝜖𝑦 = 𝜖𝑀 𝜖𝑨 𝜖𝑨 𝜖𝑦 𝜖𝑀 𝜖𝑨 𝜖𝑨 𝜖𝑦 𝜖𝑨 𝜖𝑧 𝑔

Gradient Descent for Neural Networks

39 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦0 𝑦1 𝑦2 ℎ0 ℎ1 ℎ2 ℎ3 ො 𝑧0 ො 𝑧1 𝑧0 𝑧1 ො 𝑧𝑗 = 𝐵(𝑐1,𝑗 + ෍

𝑘

ℎ𝑘𝑥1,𝑗,𝑘) ℎ𝑘 = 𝐵(𝑐0,𝑘 + ෍

𝑙

𝑦𝑙𝑥0,𝑘,𝑙) Loss function 𝑀𝑗 = ො 𝑧𝑗 − 𝑧𝑗 2 Just simple: 𝐵 𝑦 = max(0, 𝑦)

Gradient Descent for Neural Networks

40 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦0 𝑦1 𝑦2 ℎ0 ℎ1 ℎ2 ℎ3 ො 𝑧0 ො 𝑧1 𝑧0 𝑧1

ො 𝑧𝑗 = 𝐵(𝑐1,𝑗 + ෍

𝑘

ℎ𝑘𝑥1,𝑗,𝑘) ℎ𝑘 = 𝐵(𝑐0,𝑘 + ෍

𝑙

𝑦𝑙𝑥0,𝑘,𝑙) 𝑀𝑗 = ො 𝑧𝑗 − 𝑧𝑗 2

Backpropagation

𝜖𝑀𝑗 𝜖𝑥1,𝑗,𝑘 = 𝜖𝑀𝑗 𝜖 ො 𝑧𝑗 ⋅ 𝜖 ො 𝑧𝑗 𝜖𝑥1,𝑗,𝑘 𝜖𝑀𝑗 𝜖𝑥0,𝑘,𝑙 = 𝜖𝑀𝑗 𝜖 ො 𝑧𝑗 ⋅ 𝜖 ො 𝑧𝑗 𝜖ℎ𝑘 ⋅ 𝜖ℎ𝑘 𝜖𝑥0,𝑘,𝑙 Just go through layer by layer

𝜖𝑀𝑗 𝜖 ො 𝑧𝑗 = 2(ො 𝑧i − yi)

𝜖 ො 𝑧𝑗 𝜖𝑥1,𝑗,𝑘 = ℎ𝑘

if > 0, else 0

…

Gradient Descent for Neural Networks

41 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦0 𝑦1 𝑦2 ℎ0 ℎ1 ℎ2 ℎ3 ො 𝑧0 ො 𝑧1 𝑧0 𝑧1

ො 𝑧𝑗 = 𝐵(𝑐1,𝑗 + ෍

𝑘

ℎ𝑘𝑥1,𝑗,𝑘) ℎ𝑘 = 𝐵(𝑐0,𝑘 + ෍

𝑙

𝑦𝑙𝑥0,𝑘,𝑙) 𝑀𝑗 = ො 𝑧𝑗 − 𝑧𝑗 2

How many unknown weights?

• Output layer: 2 ⋅ 4 + 2
• Hidden Layer: 4 ⋅ 3 + 4

#neurons ∙ #input channels + #biases Note that some activations have also weights

Derivatives of Cross Entropy Loss

42 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦0 𝑦1 𝑦2 ℎ0 ℎ1 ℎ2 ℎ3 ො 𝑧0 ො 𝑧1 𝑧0 𝑧1

𝑀 = − ෍

𝑗=1 𝑜𝑝𝑣𝑢

(𝑧𝑗 log ො 𝑧𝑗 + 1 − 𝑧𝑗 log(1 − ො 𝑧𝑗)) ො 𝑧𝑗 = 1 1 + 𝑓−𝑡𝑗 𝑡𝑗 = ෍

j

ℎ𝑘𝑥

𝑘𝑗

• utput

scores Binary Cross Entropy loss Gradients of weights of last layer: 𝜖𝑀𝑗 𝜖𝑥

𝑘𝑗

= 𝜖𝑀𝑗 𝜖 ො 𝑧𝑗 ∙ 𝜖 ො 𝑧𝑗 𝜖𝑡𝑗 ∙ 𝜖𝑡𝑗 𝜖𝑥

𝑘𝑗

𝜖𝑀𝑗 𝜖 ො 𝑧𝑗 = −𝑧𝑗 ො 𝑧𝑗 + 1 − 𝑧𝑗 1 − ො 𝑧𝑗 = ො 𝑧𝑗 − 𝑧𝑗 ො 𝑧𝑗(1 − ො 𝑧𝑗) , 𝜖 ො 𝑧𝑗 𝜖𝑡𝑗 = ො 𝑧𝑗 1 − ො 𝑧𝑗 , 𝜖𝑡𝑗 𝜖𝑥

𝑘𝑗

= ℎ𝑘 𝜖𝑀𝑗 𝜖𝑡𝑗 = ො 𝑧𝑗 − 𝑧𝑗 ⟹ 𝜖𝑀𝑗 𝜖𝑥

𝑘𝑗

= (ො 𝑧𝑗 − 𝑧𝑗)ℎ𝑘,

Derivatives of Cross Entropy Loss

43 I2DL: Prof. Niessner, Prof. Leal-Taixé

Gradients of weights of first layer:

𝜖𝑀 𝜖𝑡

𝑘 1 = ෍ 𝑗=1 𝑜𝑝𝑣𝑢 𝜖𝑀

𝜖𝑡𝑗 𝜖𝑡𝑗 𝜖ℎ𝑘 𝜖ℎ𝑘 𝜖𝑡

𝑘 1 = ෍ 𝑗=1 𝑜𝑝𝑣𝑢

(ො 𝑧𝑗 − 𝑧𝑗)𝑥

𝑘𝑗(ℎ𝑘 1 − ℎ𝑘 )

𝜖𝑀 𝜖ℎ𝑘 = ෍

𝑗=1 𝑜𝑝𝑣𝑢 𝜖𝑀

𝜖 ො 𝑧𝑗 𝜖 ො 𝑧𝑗 𝜖𝑡𝑘 𝜖𝑡

𝑘

𝜖ℎ𝑘 = ෍

𝑗=1 𝑜𝑝𝑣𝑢 𝜖𝑀

𝜖 ො 𝑧𝑗 ො 𝑧𝑗 1 − ො 𝑧𝑗 𝑥

𝑘𝑗 = ෍ 𝑗=1 𝑜𝑝𝑣𝑢

ො 𝑧𝑗 − 𝑧𝑗 𝑥

𝑘𝑗

𝜖𝑀 𝜖𝑥𝑙𝑘

1 = ෍ 𝑗=1 𝑜𝑝𝑣𝑢 𝜖𝑀

𝜖𝑡

𝑘 1

𝜖𝑡

𝑘 1

𝜖𝑥𝑙𝑘

1 = ෍ 𝑗=1 𝑜𝑝𝑣𝑢

ො 𝑧𝑗 − 𝑧𝑗 𝑥

𝑘𝑗 ℎ𝑘 1 − ℎ𝑘

𝑦𝑙

Back to Compute Graphs & NNs

• Inputs 𝒚 and targets 𝒛
• Two-layer NN for regression

with ReLU activation

• Function we want to
• ptimize:

෍

𝑗=1 𝑜

𝑥2max 0, 𝑥1𝑦𝑗 − 𝑧𝑗

2 2

I2DL: Prof. Niessner, Prof. Leal-Taixé 44

𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 𝜏 𝑨

max(0,∙)

ො 𝑧

Gradient Descent for Neural Networks

45 I2DL: Prof. Niessner, Prof. Leal-Taixé

Initialize 𝑦 = 1, 𝑧 = 0, 𝑥1 = 1

3, 𝑥2 = 2

𝑀 𝒛, ෝ 𝒛; 𝜾 =

1 𝑜 σ𝑗 𝑜 | ො

𝑧𝑗 − 𝑧𝑗| 2

2

In our case 𝑜, 𝑒 = 1: 𝑀 = ො 𝑧 − 𝑧 2 ⇒ 𝜖𝑀

𝜖 ො 𝑧 = 2(ො

𝑧 − 𝑧) ො 𝑧 = 𝑥2 ∙ 𝜏 ⇒ 𝜖 ො

𝑧 𝜖𝑥2 = 𝜏

Backpropagation 𝜖𝑀 𝜖𝑥2 = 𝜖𝑀 𝜖 ො 𝑧 ⋅ 𝜖 ො 𝑧 𝜖𝑥2 𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3

2

2 3 4 9

1 𝜏 𝑨

max(0,∙)

ො 𝑧

1 3

Initialize 𝑦 = 1, 𝑧 = 0, 𝑥1 = 1

3, 𝑥2 = 2

𝑀 𝒛, ෝ 𝒛; 𝜾 =

1 𝑜 σ𝑗 𝑜 | ො

𝑧𝑗 − 𝑧𝑗| 2

2

In our case 𝑜, 𝑒 = 1: 𝑀 = ො 𝑧 − 𝑧 2 ⇒ 𝜖𝑀

𝜖 ො 𝑧 = 2(ො

𝑧 − 𝑧) ො 𝑧 = 𝑥2 ∙ 𝜏 ⇒ 𝜖 ො

𝑧 𝜖𝑥2 = 𝜏

Gradient Descent for Neural Networks

46 I2DL: Prof. Niessner, Prof. Leal-Taixé

Backpropagation 𝜖𝑀 𝜖𝑥2 = 𝜖𝑀 𝜖 ො 𝑧 ⋅ 𝜖 ො 𝑧 𝜖𝑥2 𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3

2

2 3 4 9

1

4 3

𝜏 𝑨 2 ∙ 2

3

max(0,∙)

ො 𝑧

1 3

Initialize 𝑦 = 1, 𝑧 = 0, 𝑥1 = 1

3, 𝑥2 = 2

𝑀 𝒛, ෝ 𝒛; 𝜾 =

1 𝑜 σ𝑗 𝑜 | ො

𝑧𝑗 − 𝑧𝑗| 2

2

In our case 𝑜, 𝑒 = 1: 𝑀 = ො 𝑧 − 𝑧 2 ⇒ 𝜖𝑀

𝜖 ො 𝑧 = 2(ො

𝑧 − 𝑧) ො 𝑧 = 𝑥2 ∙ 𝜏 ⇒ 𝜖 ො

𝑧 𝜖𝑥2 = 𝜏

Gradient Descent for Neural Networks

47 I2DL: Prof. Niessner, Prof. Leal-Taixé

Backpropagation 𝜖𝑀 𝜖𝑥2 = 𝜖𝑀 𝜖 ො 𝑧 ⋅ 𝜖 ො 𝑧 𝜖𝑥2 𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3

2

2 3 4 9

1

4 3

𝜏 𝑨

4 9 4 9

2 ∙ 2

3 ∙ 1 3

max(0,∙)

ො 𝑧

1 3

Gradient Descent for Neural Networks

48 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3 2 3 4 9

1 Backpropagation 𝜖𝑀 𝜖𝑥1 = 𝜖𝑀 𝜖 ො 𝑧 ⋅ 𝜖 ො 𝑧 𝜖𝜏 ⋅ 𝜖𝜏 𝜖𝑨 ⋅ 𝜖𝑨 𝜖𝑥1 𝜏 𝑨

max(0,∙)

4 9 4 9

2 ො 𝑧

1 3

Initialize 𝑦 = 1, 𝑧 = 0, 𝑥1 = 1

3, 𝑥2 = 2

In our case 𝑜, 𝑒 = 1: 𝑀 = ො 𝑧 − 𝑧 2 ⇒ 𝜖𝑀

𝜖 ො 𝑧 = 2(ො

𝑧 − 𝑧) ො 𝑧 = 𝑥2 ∙ 𝜏 ⇒ 𝜖 ො

𝑧 𝜖𝜏 = 𝑥2

𝜏 = max 0, 𝑨 ⇒ 𝜖𝜏

𝜖𝑨 = ቊ1 if 𝑦 > 0

0 else 𝑨 = 𝑦 ∙ 𝑥1 ⇒

𝜖𝑨 𝜖𝑥1 = 𝑦

Initialize 𝑦 = 1, 𝑧 = 0, 𝑥1 = 1

3, 𝑥2 = 2

In our case 𝑜, 𝑒 = 1: 𝑀 = ො 𝑧 − 𝑧 2 ⇒ 𝜖𝑀

𝜖 ො 𝑧 = 2(ො

𝑧 − 𝑧) ො 𝑧 = 𝑥2 ∙ 𝜏 ⇒ 𝜖 ො

𝑧 𝜖𝜏 = 𝑥2

𝜏 = max 0, 𝑨 ⇒ 𝜖𝜏

𝜖𝑨 = ቊ1 if 𝑦 > 0

0 else 𝑨 = 𝑦 ∙ 𝑥1 ⇒

𝜖𝑨 𝜖𝑥1 = 𝑦

Gradient Descent for Neural Networks

49 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3 2 3 4 9

1 Backpropagation 𝜖𝑀 𝜖𝑥1 = 𝜖𝑀 𝜖 ො 𝑧 ⋅ 𝜖 ො 𝑧 𝜖𝜏 ⋅ 𝜖𝜏 𝜖𝑨 ⋅ 𝜖𝑨 𝜖𝑥1

4 3

𝜏 𝑨 2 ∙ 2

3

max(0,∙)

4 9 4 9

2

4 3

ො 𝑧

1 3

Initialize 𝑦 = 1, 𝑧 = 0, 𝑥1 = 1

3, 𝑥2 = 2

In our case 𝑜, 𝑒 = 1: 𝑀 = ො 𝑧 − 𝑧 2 ⇒ 𝜖𝑀

𝜖 ො 𝑧 = 2(ො

𝑧 − 𝑧) ො 𝑧 = 𝑥2 ∙ 𝜏 ⇒ 𝜖 ො

𝑧 𝜖𝜏 = 𝑥2

𝜏 = max 0, 𝑨 ⇒ 𝜖𝜏

𝜖𝑨 = ቊ1 if 𝑦 > 0

0 else 𝑨 = 𝑦 ∙ 𝑥1 ⇒

𝜖𝑨 𝜖𝑥1 = 𝑦

Gradient Descent for Neural Networks

50 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3 2 3 4 9

1 Backpropagation 𝜖𝑀 𝜖𝑥1 = 𝜖𝑀 𝜖 ො 𝑧 ⋅ 𝜖 ො 𝑧 𝜖𝜏 ⋅ 𝜖𝜏 𝜖𝑨 ⋅ 𝜖𝑨 𝜖𝑥1

4 3 8 3

𝜏 𝑨 2 ∙ 2

3 ∙ 2

max(0,∙)

4 9 4 9

2 ො 𝑧

1 3

Initialize 𝑦 = 1, 𝑧 = 0, 𝑥1 = 1

3, 𝑥2 = 2

In our case 𝑜, 𝑒 = 1: 𝑀 = ො 𝑧 − 𝑧 2 ⇒ 𝜖𝑀

𝜖 ො 𝑧 = 2(ො

𝑧 − 𝑧) ො 𝑧 = 𝑥2 ∙ 𝜏 ⇒ 𝜖 ො

𝑧 𝜖𝜏 = 𝑥2

𝜏 = max 0, 𝑨 ⇒ 𝜖𝜏

𝜖𝑨 = ቊ1 if 𝑦 > 0

0 else 𝑨 = 𝑦 ∙ 𝑥1 ⇒

𝜖𝑨 𝜖𝑥1 = 𝑦

Gradient Descent for Neural Networks

51 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3 2 3 4 9

1 Backpropagation 𝜖𝑀 𝜖𝑥1 = 𝜖𝑀 𝜖 ො 𝑧 ⋅ 𝜖 ො 𝑧 𝜖𝜏 ⋅ 𝜖𝜏 𝜖𝑨 ⋅ 𝜖𝑨 𝜖𝑥1

4 3 8 3 8 3

𝜏 𝑨 2 ∙ 2

3 ∙ 2 ∙ 1

max(0,∙)

4 9 4 9

2 ො 𝑧

1 3

Initialize 𝑦 = 1, 𝑧 = 0, 𝑥1 = 1

3, 𝑥2 = 2

In our case 𝑜, 𝑒 = 1: 𝑀 = ො 𝑧 − 𝑧 2 ⇒ 𝜖𝑀

𝜖 ො 𝑧 = 2(ො

𝑧 − 𝑧) ො 𝑧 = 𝑥2 ∙ 𝜏 ⇒ 𝜖 ො

𝑧 𝜖𝜏 = 𝑥2

𝜏 = max 0, 𝑨 ⇒ 𝜖𝜏

𝜖𝑨 = ቊ1 if 𝑦 > 0

0 else 𝑨 = 𝑦 ∙ 𝑥1 ⇒

𝜖𝑨 𝜖𝑥1 = 𝑦

Gradient Descent for Neural Networks

52 I2DL: Prof. Niessner, Prof. Leal-Taixé

𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3 1 3 2 3 4 9

1 Backpropagation 𝜖𝑀 𝜖𝑥1 = 𝜖𝑀 𝜖 ො 𝑧 ⋅ 𝜖 ො 𝑧 𝜖𝜏 ⋅ 𝜖𝜏 𝜖𝑨 ⋅ 𝜖𝑨 𝜖𝑥1

4 3 8 3 8 3 8 3 8 3

𝜏 𝑨 2 ∙ 2

3 ∙ 2 ∙ 1

∙ 1

max(0,∙)

4 9 4 9

2 ො 𝑧

Gradient Descent for Neural Networks

53 I2DL: Prof. Niessner, Prof. Leal-Taixé

• Function we want to
• ptimize:
• Computed gradients wrt

to weights 𝑥1and 𝑥2

• Now: update the weights

𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧 𝑀 1

1 3 1 3 1 3 2 3 4 9

1

4 3 8 3 8 3 8 3 8 3

𝜏 𝑨

max(0,∙)

4 9 4 9

2 𝒙′ = 𝒙 − 𝛽 ∙ 𝛼

𝒙𝑔 = 𝑥1

𝑥2 − 𝛽 ∙ 𝛼

𝑥1𝑔

𝛼

𝑥2𝑔

=

1 3

2 − 𝛽 ∙

8 3 4 9

𝑔 𝑦, 𝒙 = ෍

𝑗=1 𝑜

𝑥2max 0, 𝑥1𝑦𝑗 − 𝑧𝑗

2 2

But: how to choose a good learning rate 𝛽 ?

ො 𝑧

Gradient Descent

• How to pick good learning rate?
• How to compute gradient for single training pair?
• How to compute gradient for large training set?
• How to speed things up? More to see in next

lectures…

54 I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization

55 I2DL: Prof. Niessner, Prof. Leal-Taixé

Recap: Basic Recipe for ML

• Split your data

56 I2DL: Prof. Niessner, Prof. Leal-Taixé

Find your hyperparameters train test validation 20% 60% 20%

Other splits are also possible (e.g., 80%/10%/10%)

Over- and Underfitting

57 I2DL: Prof. Niessner, Prof. Leal-Taixé

Underfitted Appropriate Overfitted

Source: Deep Learning by Adam Gibson, Josh Patterson, O‘Reily Media Inc., 2017

Training a Neural Network

• Training/ Validation curve

58 I2DL: Prof. Niessner, Prof. Leal-Taixé

How can we prevent our model from overfitting? Regularization

Credits: Deep Learning. Goodfellow et al.

Training error too high Generalization gap is too big

Regularization

• Loss function 𝑀(𝒛, ෝ

𝒛, 𝜾) = σ𝑗=1

𝑜

ො 𝑧𝑗 − 𝑧𝑗 2 + 𝜇𝑆(𝜾)

• Regularization techniques

– L2 regularization – L1 regularization – Max norm regularization – Dropout – Early stopping – ...

59 I2DL: Prof. Niessner, Prof. Leal-Taixé

Add regularization term to loss function More details later

Regularization: Example

• Input: 3 features 𝒚 = [1, 2, 1]
• Two linear classifiers that give the same result:
• 𝜄1 = 0, 0.75, 0
• 𝜄2 = [0.25, 0.5, 0.25]

60 I2DL: Prof. Niessner, Prof. Leal-Taixé

Ignores 2 features Takes information from all features

Regularization: Example

• Loss 𝑀 𝒛, ෝ

𝒛, 𝜾 = σ𝑗=1

𝑜

𝑦𝑗𝜄

𝑘𝑗 − 𝑧𝑗 2 + 𝜇𝑆 𝜾

• L2 regularization

𝜄1 0 + 0.752 + 0 = 0.5625 𝜄2 0.252 + 0.52 + 0.252 = 0.375 𝑦 = 1, 2, 1 , 𝜄1 = 0, 0.75, 0 , 𝜄2 = [0.25, 0.5, 0.25]

61

Minimization

𝑆 𝜾 = ෍

𝑗=1 𝑜

𝜄𝑗

2

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization: Example

• Loss 𝑀 𝒛, ෝ

𝒛, 𝜾 = σ𝑗=1

𝑜

𝑦𝑗𝜄

𝑘𝑗 − 𝑧𝑗 2 + 𝜇𝑆 𝜾

• L1 regularization

𝜄1 0 + 0.75 + 0 = 0.75 𝜄2 0.25 + 0.5 + 0.25 = 1 𝑦 = 1, 2, 1 , 𝜄1 = 0, 0.75, 0 , 𝜄2 = [0.25, 0.5, 0.25]

62

Minimization

𝑆 𝜾 = ෍

𝑗=1 𝑜

𝜄𝑗

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization: Example

• Input: 3 features 𝒚 = [1, 2, 1]
• Two linear classifiers that give the same result:

𝜄1 = 0, 0.75, 0 𝜄2 = [0.25, 0.5, 0.25]

63

Ignores 2 features Takes information from all features

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization: Example

• Input: 3 features 𝒚 = [1, 2, 1]
• Two linear classifiers that give the same result:

𝜄1 = 0, 0.75, 0 𝜄2 = [0.25, 0.5, 0.25]

64

L1 regularization enforces sparsity Takes information from all features

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization: Example

• Input: 3 features 𝒚 = [1, 2, 1]
• Two linear classifiers that give the same result:

𝜄1 = 0, 0.75, 0 𝜄2 = [0.25, 0.5, 0.25]

65

L1 regularization enforces sparsity L2 regularization enforces that the weights have similar values

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization: Effect

• Dog classifier takes different inputs

66

Furry Has two eyes Has a tail Has paws Has two ears L1 regularization will focus all the attention to a few key features

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization: Effect

• Dog classifier takes different inputs

67

Furry Has two eyes Has a tail Has paws Has two ears L2 regularization will take all information into account to make decisions

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization for Neural Networks

68

Combining nodes: Network output + L2-loss + regularization 𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧

max(0,∙)

+ 𝑀

𝑀2 loss

𝑆(𝑥1, 𝑥2) ∙ 𝜇 ෍

𝑗=1 𝑜

𝑥2max 0, 𝑥1𝑦𝑗 − 𝑧𝑗

2 2 + 𝜇𝑆 𝑥1, 𝑥2

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization for Neural Networks

69

Combining nodes: Network output + L2-loss + regularization 𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧

max(0,∙)

+ 𝑀

𝑀2 loss

𝑆(𝑥1, 𝑥2) ∙ 𝜇 ෍

𝑗=1 𝑜

𝑥2max 0, 𝑥1𝑦𝑗 − 𝑧𝑗

2 2 + 𝜇

𝑥1 𝑥2

2 2

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization for Neural Networks

70

Combining nodes: Network output + L2-loss + regularization 𝑦 𝑥1 ∗ 𝑥2 ∗ 𝑧

max(0,∙)

+ 𝑀

𝑀2 loss

𝑆(𝑥1, 𝑥2) ∙ 𝜇 ෍

𝑗=1 𝑜

𝑥2max 0, 𝑥1𝑦𝑗 − 𝑧𝑗

2 2 + 𝜇(𝑥1 2 + 𝑥2 2)

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization

71

Credits: University of Washington

What is the goal of regularization? What happens to the training error?

Decision Boundary Regularization 𝜇 = 0.001 𝜇 = 000001 𝜇 = 0 𝜇 = 1 𝜇 = 10

I2DL: Prof. Niessner, Prof. Leal-Taixé

Regularization

• Any strategy that aims to

72

Lower validation error Increasing training error

I2DL: Prof. Niessner, Prof. Leal-Taixé

Next Lecture

• This week:

– Check exercises – Check office hours ☺

• Next lecture

– Optimization of Neural Networks – In particular, introduction to SGD (our main method!)

73 I2DL: Prof. Niessner, Prof. Leal-Taixé

See you next week ☺

74 I2DL: Prof. Niessner, Prof. Leal-Taixé

Further Reading

• Backpropagation

– Chapter 6.5 (6.5.1 - 6.5.3) in http://www.deeplearningbook.org/contents/mlp.html – Chapter 5.3 in Bishop, Pattern Recognition and Machine Learning – http://cs231n.github.io/optimization-2/

• Regularization

– Chapter 7.1 (esp. 7.1.1 & 7.1.2) http://www.deeplearningbook.org/contents/regularization.html – Chapter 5.5 in Bishop, Pattern Recognition and Machine Learning

75 I2DL: Prof. Niessner, Prof. Leal-Taixé